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Overview 


Thi s report summarizes the two-year findings of a rigorous random assignment evaluation of the 
WorkAdvance model, a sectoral training and advancement initiative. Launched in 2011, WorkAd- 
vance goes beyond the previous generation of employment programs by introducing demand-driven 
skills training and a focus on jobs that have career pathways. The model is heavily influenced by the 
positive findings from the Sectoral Employment Impact Study (SE1S) completed in 2010. A major 
component of the WorkAdvance model, in common with the programs studied in the SE1S, is 
formal training offering industry-recognized certifications, reflecting the hypothesis that skill s 
acquisition is necessary for advancement. The model also requires providers to be far more employer- 
facing than traditional training programs, taking into account multiple employers’ changing skill 
requirements, employee assessment practices, and personnel needs. This report presents the imple- 
mentation, cost, participation, and two-year economic impacts of WorkAdvance. The economic 
results are based on unemployment insurance earnings records and a second-year follow-up survey. 

The WorkAdvance program operations and evaluation are funded through the federal Social 
Innovation Fund (S1F), a public -private partnership administered by the Coiporation for National 
and Community Service. This S1F project is led by the Mayor’s Fund to Advance New York City 
and the NYC Center for Economic Opportunity in collaboration with MDRC. 

Key Findings 

• All providers translated the WorkAdvance model into a set of concrete services, but it took time 
— more than a year for some components and providers — and a substantial amount of tech- 
nical assistance and support. As a result, at some sites, later study enrol Ices were more likely 
than earlier ones to experience a fully implemented and “mature” WorkAdvance program. 

• Overall, WorkAdvance resulted in large increases in participation in every category of services, 
as well as in training completion, credential acquisition, and employment in the targeted sector, 
compared with what would have happened in the absence of the program. Expenditures for the 
operation of WorkAdvance fell between $5,200 and $6,700 per participant at the four providers 
delivering the program. 

• WorkAdvance providers increased earnings, with variation in results that closely matched the 
providers’ experience in running sector-based programs and the extent to which the services 
they offered were demand driven. The most experienced sectoral provider, Per Scholas, had 
large and consistent impacts on both primary and secondary outcomes. Madison Strategies 
Group and Towards Employment, providers new to sectoral training, had promising but less 
consistent results that grew stronger for later enrollees. One provider, St. Nicks Alliance, did not 
produce positive impacts. The results did not differ dramatically across subgroups, though en- 
couragingly, WorkAdvance was able to increase earnings among the long-term unemployed. 

The evaluation as a whole provides important information for workforce development providers 
interested in pursuing a sector strategy. The analysis considers the role played by providers’ sector- 
specific training and preparation and the role played by the nature of the sectors themselves. Future 
priorities that emerge from the results are (1) understanding how to help the more disadvantaged access 
the programs and (2) learning how to build service capacity, given how complex the model is to run. 
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Preface 


In recent years, the workforce development field has increasingly recognized that programs 
must work closely with employers to more effectively serve the needs of job seekers. The 
sectoral programs that have emerged bring together employers, job seekers, community 
colleges, and other stakeholders to adopt coordinated regional economic and workforce devel- 
opment strategies that target industries and occupations with a growing demand for workers as 
well as the career ladders that afford opportunities for upward mobility. The aim of this coordi- 
nation is both to promote upward mobility and to spur economic development by increasing the 
skills of the local workforce, in turn attracting and retaining employers. 

While the sectoral strategy has been around for many years, rigorous evidence on its 
potential benefits has emerged only recently, most notably in Public/Private Ventures’ Sectoral 
Employment Impact Study, completed in 2010. The study excited policymakers by showing 
that the sectoral strategy can increase earnings. Yet it left many questions unanswered; the 
samples were small and the data sources relatively limited. Moreover, the providers in the study 
were well established, mature organizations. Could those same results be achieved with a 
broader range of providers and in a different economy? 

From this convergence of evidence and policy came WorkAdvance, which combined 
the key elements of the Sectoral Employment Impact programs and the best of what was known 
about advancement programs. The results, shown in this report, are encouraging. Together the 
four WorkAdvance sites helped participants earn an average of 14 percent (or nearly $2,000 in 
annual income) more than they otherwise would have earned two years after they entered the 
program. The effects differed by site, ranging from no earnings gain at one provider to a 26 
percent increase at the most effective provider. These results confirm that sectoral programs, 
when well implemented, can increase earnings among low-income individuals. 

It is important to note that the benefits to participants can take at least a year to emerge 
for experienced providers; for those new to the sectoral strategy, it will take longer. The results 
also show that sector programs must continually monitor developments in the industries they 
target and adapt as demand changes. An important priority for the field should be developing 
guidance to help providers adjust to a dual-customer, demand-driven approach. 

As local Workforce Investment Boards and city and state leaders work to implement 
the Workforce Innovation and Opportunity Act (WIOA), with its strong sectoral focus, they 
should consider these findings and draw upon the lessons herein. The challenge will be how to 
incorporate the WorkAdvance approach into larger workforce systems without losing the local 
focus inherent in sector programs. 

Gordon L. Berlin 
President 
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Executive Summary 


Even in good economic times, many adults — particularly individuals with no more than a high 
school education — struggle to obtain and maintain jobs that pay enough to support their fami- 
lies and permit upward mobility. At the same time, some employers report difficulty finding 
employees with the right skills to meet their needs. This report summarizes the two-year effec- 
tiveness results of WorkAdvance — a workforce development program that seeks to meet the 
needs of participants and employers equally. For unemployed and low- wage working adults, the 
program provides occupational skills training in targeted sectors that have good quality jobs and 
room for advancement within established career pathways. The program’s strategy is a sector- 
based one, in which program management and staff members also seek to fully understand — 
and fulfill — the skill requirements and other needs of employers in the targeted sectors. 

In brief, the two-year WorkAdvance effectiveness findings are encouraging: A rigor- 
ous randomized controlled trial indicates that sectoral programs can increase earnings among 
low-income individuals. One provider had large and consistent impacts on both primary and 
secondary outcomes. Two providers, both of which were new to the sectoral strategy and 
gained experience over time, had promising, though less consistent, results that grew stronger 
for later entrants into the program. One provider did not produce positive impacts. The results 
also show that these types of programs are hard to run well, and, even when they are imple- 
mented well, impacts take time to emerge. The evaluation as a whole provides important in- 
formation for workforce development providers interested in pursuing a sector strategy. The 
analysis considers the role played in the results by providers preparing participants for jobs in 
specific sectors and the role played by the nature of the sectors themselves. 

WorkAdvance has been implemented in diverse settings by four providers specializing 
in specific sectors in which they have sought to develop relationships with employers and in- 
depth industry knowledge: Per Scholas (in New York City) targeted the information technology 
(IT) sector; St. Nicks Alliance (also in New York City) focused on environmental remediation; 
Madison Strategies Group (in Tulsa, Oklahoma) focused on transportation and, later, manufac- 
turing; and Towards Employment (in northeast Ohio) targeted health care and manufacturing. 
Table ES.l presents information on the types of training provided at each site. The providers 
differed in a variety of ways. Two of the providers (Per Scholas and Madison Strategies Group) 
were solely focused on WorkAdvance and other sectoral training initiatives. By contrast, St. 
Nicks Alliance is a large multiservice organization with a small workforce division. Towards 
Employment runs a comprehensive employment program in addition to the WorkAdvance pro- 
gram, providing job placement and other services. The providers also differ significantly with 
regard to their experience. Per Scholas has been operating an IT sector program since 1998, and 


ES-1 



The WorkAdvance Study 
Table ES.l 


Key Details of WorkAdvance Service Providers 


Site 

(Location) 

Per Scholas 
(Bronx, NY) 

St. Nicks Alliance 
(Brooklyn, NY) 

Madison Strategies 
Group 
(Tulsa, OK) 

Towards Employment 
(Cleveland, OH) 

Targeted 

sector(s) 

Information technology 

Environmental 

remediation 

Transportation, 

manufacturing 

Health care, 
manufacturing 


690 

479 

697 

698 

Structure 

Single program 

Division of large 
multiservice program 

Single program 

Multiple employment 
programs 

Approach 

Training- first 

Training-first 

Mixed until fall 2012 
(training-first and 
placement-first) and 
later predominantly 
training-first 

Mixed until fall 2012 
(training-first and 
placement-first) and 
later predominantly 
training-first 

Occupational skills training 

Length 

15 weeks 

5 to 12 weeks 

4 to 32 weeks 

2 to 17 weeks 

Location 

On-site 

On-site for pest control 
training; mix of on- and 
off-site at private 
schools for other 
training courses 

Off-site at private or 
technical schools or 
community colleges 

Off-site at private or 
technical schools or 
community colleges 

Offerings 

A Plus, Network Plus 

Environmental 
Remediation, 
Commercial Driver's 

License B with hazmat 
endorsement, Pest 
Control 

Aviation 
Manufacturing, 
Commercial Driver's 
License A and B, 
Computerized 
Numerical Control 
(CNC) Machining, 
Diesel Mechanic, 
Welding, Supervisory 
Leadership 

Computerized 
Numerical Control 
(CNC) Machining, 
Welding, Phlebotomy, 
Certified Health Care 
Access Associate, 
Patient Care Assistant, 
State-Tested Nurse 
Assistant, Medical 
Billing and Coding 


SOURCES: MDRC sample size counts from the WorkAdvance baseline information form and other information 
from documentation supplied by providers and interviews with provider staff members. 
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St. Nicks Alliance has given training in environmental remediation since 2001. By contrast, 
Madison Strategies Group was operating in a new city and Towards Employment was operating 
in a new sector (manufacturing). 

WorkAdvance’s programming and evaluation have been conducted under the auspices 
of the Social In novation Fund (SIF). Administered by the Corporation for National and Com- 
munity Service, the SIF is a public-private partnership designed to identify and expand effective 
solutions to critical social challenges. WorkAdvance is part of the New York City Center for 
Economic Opportunity (CEO) 2010 SIF project, which is led by CEO and the Mayor’s Fund to 
Advance New York City in collaboration with MDRC. MDRC is leading the WorkAdvance 
evaluation; has provided technical assistance to the providers; and, jointly with CEO, has moni- 
tored providers’ operations. Funding for the WorkAdvance program and evaluation came from 
the SIF and a broad array of local funding partners. 


The WorkAdvance Program Model 

The WorkAdvance model goes beyond the previous generation of employment programs by 
introducing sector-based skills training and a focus on jobs that have career ladders or pathways, 
rather than just seeking “any job” or an initial credential for participants. The model is heavily 
influenced by the positive findings from the Sectoral Employment Impact Study (SEIS) com- 
pleted in 2010. 1 A major component of the WorkAdvance model, in common with the programs 
studied in the SEIS, is formal training offering industry-recognized certifications, reflecting the 
hypothesis that skills acquisition is necessary for advancement. Additionally, the WorkAdvance 
model requires providers to be far more employer- facing than traditional training programs, tak- 
ing into account multiple employers’ changing skill requirements, employee assessment prac- 
tices, and human resource needs. 

The essential theory behind WorkAdvance is that strategic, demand-guided upgrades in 
human capital — that is, education and employment-related skills and experience — will even- 
tually lead to advancement in the labor market. This theory informs the key components of the 
WorkAdvance model: 

1. Intensive screening of program applicants before enrollment — a practice 
not always found in employment programs offered to low-income individu- 
als — intended to ensure that the program providers select participants who 
can take advantage of the skills training in the sector and be qualified for 
specific occupations within it. 


’Sheila Maguire, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz, Tuning In to 
Local Labor Markets: Findings from the Sectoral Employment Study (Philadelphia: Public/Private Ven- 
tures, 2010). 
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2. Sector-appropriate preemployment and career readiness services, consist- 
ing of an orientation to the sector, career readiness training (which stresses 
how to search for work in the sector and how to comport oneself on the job), 
individualized career coaching, and limited supportive services to sustain 
program engagement and assist participants to complete their training and 
find employment. 

3. Sector-specific occupational skills training, intended to impart skills and 
lead to credentials that will substantially enhance workers’ employment op- 
portunities. Training is geared toward current job openings in specific sectors 
and occupations, and training offerings are to be adapted to changes in avail- 
able jobs. (In the first half of the program enrollment period, the goal was to 
place approximately half the enrollees at two providers into a placement-first 
track, in which they skipped the initial training phase, so the model’s training 
component was not offered to all program enrollees.) 

4. Sector-specific job development and placement services, intended to facili- 
tate entry into positions for which participants have been trained and for 
which there are thought to be genuine opportunities for continued skill de- 
velopment and career advancement. The providers’ job developers are to 
maintain strong relationships with employers who hire individuals with the 
kinds of skills the program imparts. 

5. Postemployment retention and advancement services, provided to assist 
participants beyond the placement stage. Providers are to maintain close con- 
tact with workers and employers to assess performance, offer coaching to 
address any complicating life situations that might arise for workers, help 
identify next-step job opportunities and skills training to enable participants 
to move up career ladders over time, and assist with rapid reemployment if 
workers lose their jobs. It should be noted that the full effects of these ser- 
vices may not yet be seen in this report given that only slightly over two 
years of follow-up data are available. 


Evaluation Design and Study Enrollee Characteristics 

The WorkAdvance programs at the four sites were studied using a random assignment design, a 
methodology often referred to as the “gold standard” in evaluation research, which allows prac- 
titioners and policymakers to have a very high degree of confidence in the results. From June 
201 1 to June 2013, individuals who met the WorkAdvance eligibility criteria, as well as the re- 
quirements for the specific sector programs, were assigned at random to the WorkAdvance 
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group or to the control group. Members of the WorkAdvance group were eligible to receive 
WorkAdvance services, while those in the control group were not eligible for these services but 
could get other services and support available in the community. Both research groups were 
tracked over time. The random assignment process ensured that, when individuals entered the 
study, there were no systematic differences in sample members’ characteristics, measured or 
unmeasured, between the two research groups. Thus, any differences between the groups that 
emerge after random assignment can be attributed to the WorkAdvance program. 

During the evaluation period, WorkAdvance targeted unemployed and low-wage work- 
ing adults with a monthly family income below 200 percent of the federal poverty level. Adults 
who were working at the time they entered the study were required to be earning less than $15 
per hour. Sector-specific requirements might include drug testing, literacy or numeracy testing, 
or screens for criminal history, and in many cases individuals had to demonstrate that they could 
engage in training full time and that they had a high degree of interest in a career in the sector. 
Across the sites, 2,564 individuals were enrolled in the study and randomly assigned to either 
the WorkAdvance group (1,293) or the control group (1,271). Site-level sample sizes range 
from 479 at the St. Nicks Alliance site up to 698 at Towards Employment. 2 

Across the sites, the average WorkAdvance sample member was 34 years old, and the 
majority of sample members were male (73 percent) and single (67 percent). (These averages 
mask important variation by site and sector. For example, the health care sector at Towards 
Employment mostly enrolled females.) Half the participants were black/African- American, 18 
percent were white, and 17 percent were Latino/Hispanic. Almost all sample members had at 
least a high school diploma or equivalent, and over half the sample had at least some college 
education, though most did not have a college degree of any type. Only one in five were work- 
ing at the time they entered the study, and more than one-third of sample members had been 
unemployed for at least seven months immediately preceding study entry. Overall, 1 5 percent of 
sample members had work experience in their targeted industry. At study entry, less than half 
the enrollees were covered by health insurance, more than one-third were receiving Supple- 
mental Nutrition Assistance Program (SNAP) benefits (food stamps), 16 percent were receiving 
unemployment insurance benefits (even though most were unemployed), and 6 percent were 
receiving Temporary Assistance for Needy Families (TANF). Despite fairly high levels of edu- 
cation and work experience, many sample members still faced substantial barriers to employ- 
ment: One-quarter of enrollees, for example, had a criminal conviction, and even higher rates 
were seen among transportation and manufacturing sector enrollees. 


‘Attrition from the study was minimal: Only 20 individuals (less than 0.8 percent of the original sample) 
withdrew from the study at any point following their random assignment. 
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This report extends the evaluation findings of two previous documents, 3 addressing a 
number of research questions: How did the four providers implement the program model, and 
what adjustments did they make over time? To what extent did WorkAdvance increase en- 
gagement in employment and training services, compared with what would have happened in 
the absence of the program (the control group)? What does it cost to operate the WorkAdvance 
program? To what extent did WorkAdvance, over a two-year follow-up period, improve indi- 
viduals’ employment, earnings, income (including public assistance), and life satisfaction? And 
what helps explain the patterns in WorkAdvance’s economic effects — for example, what is the 
influence of program maturity? 

Sources of data analyzed in this report to determine WorkAdvance’s effectiveness in- 
clude a follow-up survey, in which the average survey respondent was interviewed 22 months 
following his or her random assignment (the “Year 2 Survey,” for which about 80 percent of the 
entire study sample was interviewed), and unemployment insurance data, available for nine 
quarters (2.25 years) following random assignment (obtained for all sample enrollees). Thus, all 
study enrollees have approximately two years of common follow-up, in most cases from both 
the survey and the unemployment insurance administrative records. 


Key Findings 

Analyses in this report yielded the following key findings. 

Implementation and Participation Findings 

• Translating the WorkAdvance model into a set of concrete services took 
time — more than a year for some components and providers — and a 
substantial amount of technical assistance and related support; some 
providers required more time and assistance than others. As a result, at 
some sites, a fully implemented and “mature” WorkAdvance program 
was more likely to be experienced by later study enrollees than by early 
study enrollees. 

All four providers eventually implemented all the WorkAdvance model components, 
with postemployment services the last to be fully developed and implemented. The engagement 
of WorkAdvance group members in career readiness services, occupational skills training, and 
postemployment retention and advancement services was high for all four providers, in part re- 


3 Betsy Tessler, WorkAdvance: Testing a New’ Approach to Increase Employment Advancement for Low- 
Skilled Adults (New York: MDRC, 2013), and Betsy L. Tessler, Michael Bangser, Alexandra Pennington, 
Kelsey Schaberg, and Hannah Dalporto, Meeting the Needs of Workers and Employers: Implementation of a 
Sector-Focused Career Advancement Model for Low-Skilled Adults (New York: MDRC, 2014). 
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fleeting the pre-random assignment intake process that screened out individuals who were not 
motivated or able to complete the intake process, who did not have the literacy levels needed to 
take full advantage of the training, or who had other impediments to being hired in the sector. 

While all WorkAdvance providers delivered the model components, differences in im- 
plementation across the providers were apparent. Per Scholas’s history of providing information 
technology training and its long-standing connections with employers in that sector gave it a 
head start on the other providers. Because it did not outsource skills training but provided it on- 
site, Per Scholas staff members also had more opportunities to interact with program partici- 
pants, deliver career readiness services, and reinforce workplace behaviors. Madison Strategies 
Group and Towards Employment initially operated a dual-track approach, in which some enrol- 
lees were first placed in jobs, before being offered training opportunities. This resulted in early 
enrollees at those two sites being less likely than enrollees at the other two sites to ever start and 
complete occupational skills training. These two providers were also the newest to sectoral 
training. For these reasons, later program enrollees (the “late cohort”) probably experienced a 
stronger program at those two sites. Finally, while all providers had ties to employers in their 
targeted sectors, some providers had more experience being demand driven than others or had 
closer ties to employers. Per Scholas had the most experience delivering demand-driven ser- 
vices (which means that employer input is evident in all program components). 

• Across all sites, WorkAdvance resulted in very large increases in partic- 
ipation in every category of services, as well as in training completion 
and credential acquisition, compared with what would have happened 
in the absence of the program. 

On their own initiative the control group members obtained a range of employment ser- 
vices, including (in some cases) training in the targeted sector, from community colleges and 
other providers. It is not surprising that the control group members found their way to services, 
given that they had to be quite motivated to persist through the WorkAdvance screening pro- 
cess. WorkAdvance, however, resulted in many more individuals participating in training, as 
well as in career readiness, job search, and postemployment services, over a two-year follow-up 
period, and the evaluation is thus in a good position to measure the economic value that these 
activities and services can add. For example, WorkAdvance increased participation in vocation- 
al training in the targeted sector by approximately 40 percentage points or more at every site, 
relative to control group members’ levels (which ranged from 13 percent to 21 percent depend- 
ing on the site). WorkAdvance also increased the likelihood of completing such training by 31 
percentage points or more (as shown in Figure ES. 1) and increased the likelihood of attaining a 
vocational training credential in the targeted sector by 25 percentage points or more. These re- 
sults are impressive, particularly given that two of the providers — Madison Strategies Group 
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The WorkAdvance Study 
Figure ES.l 

Completion of Training in the Targeted Sector, by Site 



Per Scholas St. Nicks Alliance Madison Strategies Towards Employment 

Group 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


and Towards Employment — placed approximately half their early enrollees in a placement- 
first track. MDRC calculations indicate that expenditures on the operation of WorkAdvance 
were within the range of $5,200 to $6,700 per program group study participant at the four pro- 
viders delivering the program. 

Economic Impact Findings 

• The WorkAdvance programs at all providers increased participants’ 
employment in the targeted sector, but the size of the impact varied sub- 
stantially across the sites. 

Figure ES.2 shows that at all four sites WorkAdvance increased employment in the tar- 
geted sector, relative to the experiences of control group members. Of all the sites, Per Scholas 
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The WorkAdvance Study 
Figure ES.2 

Employment in the Targeted Sector at Current or Most Recent Job, by Site 



Per Scholas St. Nicks Alliance Madison Strategies Towards Employment 

Group 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


produced by far the largest impact on employment in the targeted sector: 61 percent of Work- 
Advance group members, when surveyed, reported working in the information technology sec- 
tor at their current or most recent job, a striking 41 percentage points higher than was the case 
for control group members. St. Nicks Alliance had the lowest rate of WorkAdvance group 
members who reported current or recent work in the targeted sector — 32 percent — but its im- 
pact of 12 percentage points on this measure is still statistically significant. Madison Strategies 
Group and Towards Employment both increased current or recent targeted sector employment 
by a little over 16 percentage points, even with relatively high control group rates. 
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• Impacts on earnings — from any type of job in any sector — varied 
across the sites, in a pattern that closely matched the providers’ experi- 
ence in running sector-based programs and the extent to which the ser- 
vices they offered were demand driven. 

Figure ES.3 shows impacts on earnings in Year 2 of the follow-up period. Early in the 
follow-up period, WorkAdvance group members’ earnings levels were lower than those of the 
control group owing to the well-known labor market opportunity cost of training participation. 
This situation, however, changed for three of the four providers in Year 2. At Per Scholas 
(which has long operated an information technology sector training program), WorkAdvance 
increased earnings by over $3,700 (or 26 percent) above the control group level in Year 2. At 
Madison Strategies Group (which was a spin-off from an experienced provider that sought to 
operate programs in a new area of the country) and Towards Employment (an organization 
relatively new to working in one of the sectors in which it operated for WorkAdvance), statis- 
tically significant impacts on earnings began to emerge in Year 2, particularly toward the end 
of that year. At St. Nicks Alliance, statistically significant impacts on earnings had not yet 
emerged by Year 2. 4 

• At the two sites where many early enrollees were channeled into a 
placement-first track, impacts were stronger for the late enrollee cohort 
than for the early enrollee cohort, as the providers matured in their abil- 
ity to deliver services and switched to a mostly training-first model. 

Figure ES.4 shows the impacts of WorkAdvance by random assignment cohort — an 
analysis that in the evaluation’s planning stage was seen as critically important. Two of the pro- 
grams (Madison Strategies Group and Towards Employment) both initially implemented a 
mixed model, in which the goal was to place half the program participants into jobs in the tar- 
geted sectors before training. As shown in Figure ES.4, at both providers impacts were dramati- 
cally stronger for individuals who enrolled in the second half of the intake period than for indi- 
viduals who enrolled in the first half. It is difficult to determine reliably the extent to which the 
better results for the later cohort were due to a greater emphasis on training versus an overall 
maturation of the program at these providers, both of which were new to sectoral programming. 


4 As discussed in the report, two measures were prespecified as “confirmatory” measures: earnings in the 
last quarter available for the full sample (which originally was planned as Quarter 7, but was changed to Quar- 
ter 10 after data collection was extended) and employment (in any sector) at the time of the survey. Only the 
Per Scholas program produced statistically significant impacts if just these two measures are considered. Madi- 
son Strategies Group and Towards Employment produced impacts that are not quite statistically significant on 
these measures, though the impacts on earnings in Quarter 10 are statistically significant when those two sites 
are pooled. St. Nicks Alliance did not produce impacts on either measure. 
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The WorkAdvance Study 
Figure ES.3 

Impacts on Earnings in Year 2, by Site 
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Per Scholas St. Nicks Alliance Madison Strategies Towards Employment 

Group 

SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 
State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job 
and Family Services for Towards Employment sample members; and Oklahoma Employment Security 
Commission for Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


The cohort analysis is the least relevant to Per Scholas, as it was the most mature pro- 
vider coming into the study. However, the results indicate that program effects were somewhat 
weaker for the later cohort at that site than for the early cohort; the reasons are unclear, but these 
results could relate to the strengthening local economy during the study period (and perhaps to 
other factors as well). At St. Nicks Alliance, no improvement in impacts was evident for the 
later cohort relative to the early cohort. 



ES-11 



The WorkAdvance Study 
Figure ES.4 

Impacts on Earnings in Year 2, by Site and Random Assignment Cohort 
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SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York State Department of Labor for Per Scholas 
and St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma 
Employment Security Commission for Madison Strategies Group sample members. 


NOTES: WA group = WorkAdvance (program) group. 

The early cohort includes sample members randomly assigned through Quarter 3, 2012. The late cohort includes all sample members randomly assigned 
in and after Quarter 4, 2012. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 



• The program at Per Scholas produced impacts on the widest array of 
secondary employment outcomes. However, most providers produced 
impacts on some indicators of career advancement or advancement 
potential. 

The WorkAdvance program at Per Scholas increased hourly wages and weekly earn- 
ings, and the Madison Strategies Group program increased the likelihood of participants work- 
ing at jobs with higher wages. WorkAdvance had no effect on average wages or earnings in the 
current or most recent job (as of the survey administration) at the other sites and did not affect 
the likelihood of working full time at any of the sites. 

At all sites except for St. Nicks Alliance, WorkAdvance group members were more 
likely than their control group counterparts to report that the job they held currently or most re- 
cently as of the survey administration offered many opportunities for career advancement. 
WorkAdvance also improved other aspects of job quality and advancement. Notably, WorkAd- 
vance increased the proportion of individuals who received several types of employment-related 
benefits at Madison Strategies Group; improved job satisfaction at both that site and Per Scho- 
las; and increased the proportion of people who reported working in a regular permanent job, as 
opposed to a temporary one, at Towards Employment. 

The impact analysis also examined a range of measures related to income, material 
hardship, and overall well-being. Only WorkAdvance at Per Scholas produced impacts in all 
these domains. The Per Scholas program increased income, reduced material hardship, reduced 
public assistance usage, and increased overall life satisfaction. It is unusual to see such a con- 
sistent pattern of impacts across so many domains. At the other sites, only a few impacts on 
such measures are statistically significant. 

• The extent to which WorkAdvance increased employment in targeted 
sector jobs was the critical factor in explaining the pattern of impacts 
across the sites. At all the sites, jobs in the targeted sector were generally 
of higher quality than jobs outside the targeted sector. 

For a sectoral program to produce employment and earnings impacts, the program 
needs to increase employment in the targeted sector, and the targeted sector jobs have to have 
better characteristics than jobs outside the targeted sector (that is, there needs to be a “sector 
premium”). In tenns of increasing targeted sector employment, Per Scholas was by far the most 
successful. As shown in Figure ES.2, the 41 percentage point impact on targeted sector em- 
ployment at that site was more than twice as large as the impact at any of the other sites and al- 
most four times as large as the impact at St. Nicks Alliance. This differential in targeted sector 
employment appears to explain why Per Scholas produced the strongest impacts and why St. 
Nicks Alliance produced the weakest impacts. 
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At most sites, an examination of the jobs held by WorkAdvance group members who 
worked following random assignment suggests that targeted sector jobs were better than jobs 
outside the targeted sector. At Per Scholas, St. Nicks Alliance, and Madison Strategies Group, 
targeted sector jobs paid more than other jobs that WorkAdvance group members found. In ad- 
dition, at most sites, the typical number of hours required by the jobs was higher, or other job 
characteristics, such as offered benefits, tended to be better, for jobs WorkAdvance group 
members held in the targeted sector than for jobs they held outside the targeted sector. Thus, 
there was no notable variation among the sites in the “premium” added for targeted sector jobs 
versus non-targeted sector jobs. Instead, the overriding factor for why the impacts varied from 
site to site appears to be differences in the extent to which the sites increased placements in the 
targeted sector. 

• The results also highlight that sector programs can be hard to run well. 

The WorkAdvance model was intended to go well beyond traditional workforce devel- 
opment models by incorporating a sectoral and advancement emphasis. It will take time for 
many providers to develop these capacities. In particular, the St. Nicks Alliance WorkAdvance 
program confronted numerous difficulties in adapting its more traditional vocational training 
program to the Work Advance model, which may explain why impacts have not emerged, at 
least through this report’s follow-up period. St. Nicks Alliance is a highly experienced commu- 
nity-based multiservice provider with a relatively small workforce division. The WorkAdvance 
program at St. Nicks Alliance experienced a collapse in the demand for environmental remedia- 
tion work early in the program period and faced challenges in responding to these changes. A 
more effective response would have required a more proactive approach with employers than 
St. Nicks had previously used. 

• Pooling the results from the four providers, the economic impacts of 
WorkAdvance are positive and statistically significant, but this masks 
considerable variation at the site level. 

During the evaluation’s analysis planning phase, a decision was made to focus the im- 
pact analysis at the site level, 5 given the providers’ widely varying experience with WorkAd- 
vance-like services and sectoral strategies. Still, it is instructive for policymakers to understand 
how a strategy like WorkAdvance might perform overall, across a range of providers and loca- 
tions. Figure ES.5 shows, for all the sites pooled together (left set of columns), WorkAdvance’s 
impact on earnings in the post-training period (Year 2). As a whole, WorkAdvance increased 


initially, the analysis plan called for an examination of impacts by sector. This changed after a site was 
dropped from the study, because the sector results would have lacked sufficient statistical power. The decision 
to examine impacts by site was made well before impact results were available. 
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The WorkAdvance Study 
Figure ES.5 


Impacts on Earnings in Year 2 for All Sites Pooled, 
by Prior Attachment to the Labor Market 
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SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 
State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job 
and Family Services for Towards Employment sample members; and Oklahoma Employment Security 
Commission for Madison Strategies Group sample members. 


NOTES: WA group = WorkAdvance (program) group. 

The fully attached subgroup consists of sample members who were working at random assignment or who 
were unemployed for less than one month before random assignment. The semiattached subgroup consists of 
sample members who were unemployed for one to six months before random assignment. The long-term 
unemployed subgroup consists of sample members who have never been employed or who were unemployed 
for seven or more months before random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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earnings by about $1,945, or about 14 percent, above the control group level in Year 2. The re- 
sults show that with a configuration of sites similar to those evaluated in WorkAdvance, poli- 
cymakers can expect positive impacts on earnings overall. It is important, however, to recognize 
that this overall effect will mask considerable variation at the site level. For example, without 
Per Scholas, the pooled impacts are less than half as large, though still statistically significant. 

• WorkAdvance produced positive impacts for the long-term unem- 
ployed. 

WorkAdvance operated during the long wake of the Great Recession of 2007-2009. 
During this period, the number of people who qualified as long-term unemployed increased 
markedly, and there was significant concern about the likelihood of reengaging this group in the 
labor market. Given this concern, the analysis examined WorkAdvance’s effects for subgroups 
defined by individuals’ prior level of attachment to the labor market (Figure ES.5). This anal- 
ysis found that WorkAdvance produced statistically significant impacts on employment and 
earnings for both the long-term unemployed and those who were semiattached to the labor 
market. 6 


Conclusions 

• The WorkAdvance results show that sectoral programs can increase 
earnings among low-income individuals. But even when a program is 
well implemented, the benefits take time to emerge, for providers that 
have lengthy experience in sectoral training as well as for providers that 
are new to the strategy. 

The Per Scholas program has now been shown to produce large impacts in two separate 
studies: in this study as well as in the Sector Employment Impact Study (SEIS). 7 The WorkAd- 
vance results thus provide important validation that a mature and highly functioning sectoral 
training provider can produce large impacts on a consistent basis. More generally, the results 
provide confinnation of the SEIS results, which showed that three separate providers (including 
Per Scholas) produced large effects on low-income individuals’ earnings. The WorkAdvance 


6 For this analysis, long-term unemployment was defined as never having been employed or having been 
unemployed for seven months or more immediately preceding study entry. Note that the more positive results 
among the long-term unemployed are related to the fact that sites that served more long-term unemployed 
workers had larger impacts in general. Therefore, while the results show that WorkAdvance can work for the 
long-term unemployed, the results do not necessarily imply that the program will always work better for that 
group, nor do they demonstrate that the program will be less effective among the recently or currently em- 
ployed. In general, a variety of analyses suggest that the characteristics of sample members in each site did not 
explain the variation in impacts across the sites. 

7 Maguire et al. (2010). 
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impacts at Per Scholas are evident using both survey data and unemployment insurance records 
and across a large number of primary and secondary outcomes. Moreover, the size and con- 
sistency of the impacts of Per Scholas’s sector training program are unusual in the history of 
employment and training studies. 

The WorkAdvance results also highlight that in sectoral training programs, impacts take 
time to emerge, for two reasons. First, for participants, it takes time for the training and ad- 
vancement services to be completed and for them to translate into job placements and work ad- 
vancement. Even for individuals enrolled at the experienced Per Scholas, impacts did not 
emerge until the second year of follow-up. Second, for providers new to a sector, it takes time 
for program staff relationships with employers and staff specialization in sector practices to take 
root and to result in strong and tailored training programs. Both these factors were clearly in 
play at Madison Strategies Group and Towards Employment, two providers that implemented 
WorkAdvance well, but which were new to their sector or location. At these two sites, impacts 
on participants’ earnings emerged only in the second year of follow-up. In addition, the impacts 
were much stronger for the late-enrolling cohorts, who experienced stronger programs. 

It is likely that the wide range of experience and backgrounds among the WorkAdvance 
providers is typical of the landscape of workforce development providers across the nation, and 
thus the WorkAdvance findings have important implications for Workforce Innovation and Op- 
portunity Act (WIOA) implementation and workforce practice in general. (Many of these im- 
plications are discussed in the concluding chapter of the report.) In this sense, the WorkAdvance 
evaluation provides the field with a reliable indication of how the sectoral strategy will work 
among a representative range of providers. When all sites are pooled together, the results of 
WorkAdvance are positive and statistically significant. This implies that, on average, the pro- 
grams can be effective. But the report results also show that the size and particularly the timing 
of impacts may vary critically based on where providers start from in terms of their experience 
in the sector. The sectoral strategy requires highly capable providers, and the report results made 
it clear that not all providers will produce positive impacts via the sectoral training approach. 
The results also highlight that for the sectoral and postemployment advancement strategy fea- 
tured in WorkAdvance to translate into earnings gains, providers need to focus on increasing 
targeted sector employment in jobs that have higher wages and better characteristics than the 
jobs participants can find on their own in other sectors. This, in turn, will happen only if pro- 
gram services are truly demand driven, which requires that program leaders and staff members 
at all levels understand many aspects of their targeted sector — its language, needs, metrics, and 
skill requirements. 

* * * 
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While this report presents the final WorkAdvance impact estimates under the project’s 
Social Innovation Fund grant, additional reports targeted to specific audiences are planned. Fu- 
ture publications will consider the implications of the WorkAdvance experience for practition- 
ers and for the career pathways field in general. Furthermore, longer follow-up data will be col- 
lected at the three- and five-year points. The results will provide important insights into whether 
impacts grow over time and about the effects of postemployment advancement services. A full 
benefit-cost analysis will also be conducted. 
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Chapter 1 

Introduction 

The WorkAdvance Project 

Even in good economic times, many adults in the United States have difficulty obtaining jobs 
and advancing in careers that pay enough to support their families. In particular, individuals 
with no more than a high school education have seen their wages remain flat in real tenns for 
decades, and their employment is often unsteady. 1 Training programs for low-skilled adults of- 
ten fail to prepare participants for sustained employment and upward mobility, especially if the 
programs do not lead to a marketable credential, 2 or if they do not focus on jobs in high-demand 
occupations with genuine advancement opportunities. At the same time, some employers report 
difficulty finding people with the right skills to meet their needs, even in periods of high unem- 
ployment. 3 

Amid much debate about how workforce policy should address these concerns, there is 
a continuing need for clearer evidence on the best ways to promote upward mobility. The 
WorkAdvance evaluation seeks to fill the gap in evidence by testing the effectiveness of a mod- 
el that builds on previous research and practitioners’ experience in two especially important 
areas of workforce policy: sectoral training and strategies for job retention and career advance- 
ment. 


Sectoral training focuses on preparing individuals for quality jobs that employers are 
seeking to fill in specific high-demand industries or occupational clusters. A key element of this 
approach is to address the needs of employers and workers simultaneously. Although variations 
of sectoral strategies have been used for some time, increased interest was sparked by the results 
of a rigorous random assignment evaluation 4 — Public/Private Ventures’ Sectoral Employment 
Impact Study (SEIS), 5 completed in 2010 — as well as those of other tests that used less rigor- 
ous research designs, including promising results from a study of sector centers sponsored by 
the New York City Center for Economic Opportunity (CEO) that directly influenced the design 

Misliel. Bernstein, and Shierholz (2009). 

2 Hamilton and Scrivener (2012b). 

3 Morrison et al. (2011); Holzer (2013). 

4 In a random assignment evaluation, eligible individuals who apply for a program are assigned at random 
either to receive the program’s services or not to receive them. If sample sizes are large enough, the difference 
between the two groups’ outcomes — referred to as “impacts” — can be attributed to the program, since the 
two groups were statistically alike at baseline and the only difference between them is that one group received 
program services and the other did not. A random assignment study (also known as a randomized controlled 
trial) is widely held to be the most reliable way to study a program’s effectiveness, providing the highest level 
of evidence. 

5 Maguire et al. (2010). 
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of WorkAdvance . 6 The SEIS, which examined three small programs operated by organizations 
with experience in sector- focused efforts, found substantial improvements in individuals’ em- 
ployment, earnings, and wage rates over a two-year follow-up period . 7 

A key distinction between sectoral strategies and traditional employment and training 
programs is that sector programs adopt a dual-customer approach. For example, a training pro- 
vider that trains in a specific field but does not have strong relationships with employers or in- 
dustry associations in that field would not be considered a sectoral provider under this defini- 
tion. To qualify as a sector program, an initiative must bring together multiple employers in a 
given field to collaborate on developing a qualified workforce . 8 Many training programs focus 
more on the participants and work with employers only during the job placement phase. A sec- 
tor program works with employers at every stage of programming and often invites employers 
on-site for mock interviews, to consult about curriculum design, or even to provide hands-on 
training. 


WorkAdvance also draws on lessons from efforts to improve the job retention and 
career advancement of low-skilled workers after initial job placement. Retention-and- 
advancement programs have had mixed results, but much has been learned about what is likely 
to be effective and, equally important, ineffective. Particularly relevant for WorkAdvance is the 
hypothesis that concrete postemployment support — such as coaching tied to specific career 
paths and proactive reemployment services when a participant loses a job — could help indi- 
viduals not only maintain their employment, but also continue to increase their earnings over 
time . 9 By integrating the most promising features of sectoral and retention and advancement 
strategies, the designers of WorkAdvance were hopeful that this combination of services would 
produce larger and longer-lasting effects on employment, earnings, and career paths than either 
strategy might have done on its own. While other training programs have incorporated retention 
and advancement services, the WorkAdvance study provides the first rigorous test of this com- 
bination of services. 

The WorkAdvance program operations and evaluation have been conducted under the 
auspices of the Social Innovation Fund (SIF). Administered by the Corporation for National and 
Community Service, SIF is a public -private partnership designed to identify and expand efifec- 

6 A quasi-experimental evaluation of New York City’s Workforce Transportation Career Center, which 
provides sector-focused job placement, training, and support to low-income individuals interested in new or 
higher-wage positions in the transportation sector, showed that participants were three times more likely to be 
placed in jobs than the comparison group and earned $1.90 more per hour (Henderson, MacAllum, and Kara- 
kus, 2010). 

’Because the study did not follow the research sample beyond two years, it is not clear how much longer 
these positive results would have lasted and the extent to which workers would have advanced along career 
pathways. 

8 Woolsey and Groves (2013); Conway, Blair, Dawson, and Dworak-Munoz (2007). 

9 Hamilton and Scrivener (2012a). 
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tive solutions to critical social challenges. WorkAdvance has been part of the New York City 
CEO SIF project, which is led by CEO and the Mayor’s Fund to Advance New York City in 
collaboration with MDRC. MDRC is leading the WorkAdvance evaluation; has provided tech- 
nical assistance to the local providers; and, jointly with CEO, has monitored providers’ opera- 
tions. Funding for the WorkAdvance program and evaluation came from the SIF and a broad 
array of local partners that matched the SIF funding . 10 

This report provides policymakers, practitioners, and funders with important infor- 
mation on the feasibility, two-year effects (or “impacts”), and cost of expanding and replicating 
a model of this type for low-income populations in various local contexts. WorkAdvance has 
been delivered by organizations with varying missions and experience. Notably, two of the four 
providers had no previous experience operating sector programs, and although some had job 
retention services in place, none of them had ever incorporated substantial postemployment ad- 
vancement services into their programs. The WorkAdvance providers operated in three different 
locations (New York City, northeast Ohio, and Tulsa, Oklahoma) and across multiple sectors 
(transportation, infonnation technology, environmental remediation, health care, and manufac- 
turing). Consistent with the SIF goal of testing programs on a larger scale, the local WorkAd- 
vance providers recruited twice the number of people that they could serve (since half the peo- 
ple they recruited were assigned to the control group), and the number enrolled in the program 
group was twice the number enrolled in the SEIS. The WorkAdvance providers began opera- 
tions during an especially poor economy, when low-skilled workers were experiencing extend- 
ed periods of unemployment or underemployment. In contrast, the programs that participated in 
the SEIS operated primarily in a stronger economy. 

It is important to note that two years of follow-up is probably not long enough to meas- 
ure the full benefits of the WorkAdvance programs. It may take time for the benefits of a sector- 
program placement to translate into higher wages and career advancement. Current plans call 
for the collection of additional follow-up so that WorkAdvance’s economic outcomes can be 
further measured at three and five years after each participant entered the program. Assuming 
these data can be collected, this report should be viewed as showing the interim rather than final 
impacts of the WorkAdvance model. 

The rest of this chapter provides more detail on the WorkAdvance model, the experi- 
mental sites that were chosen, the research design, the economic conditions, and the target pop- 
ulations. 


10 The Mayor’s Fund to Advance New York City and CEO led fundraising efforts for the SIF initiative, 
identifying a diverse set of national and local funders to match the federal funds for support with program op- 
erations, evaluation, and oversight activities. The WorkAdvance providers worked closely with CEO to raise 
matching funds to support program operations and to meet SIF match requirements. 
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The WorkAdvance Model 

A fundamental focus on employer input and long-term career advancement is reflected 
in each of the five WorkAdvance program elements shown in Figure 1.1 (including screening, 
which precedes enrollment). WorkAdvance is a workforce development model designed to help 
unemployed and low-wage working adults increase their employment and earnings by finding 
good quality jobs in selected sectors that have room for advancement within established career 
pathways. The essential theory behind WorkAdvance is that strategic upgrades in human capital 
— that is, education and employment-related skills and experience — will lead to advancement 
in the labor market, but only if training and job preparation are directly aligned with specific job 
openings. This theoretical pathway applies to most program participants in the evaluation. For 
those who did not attend training (because they already had the requisite skills and experience) 
the theory is that strong connections to the labor market will help facilitate access to jobs in the 
selected industry. The theory also assumes that well-regarded providers with strong labor mar- 
ket connections will have a spillover effect: They will help individuals who would otherwise not 
be as well represented in the labor market. For example, sector programs can advocate for indi- 
viduals who have trouble entering the labor market because of discrimination by race or gender. 
A critical part of the sectoral vision is that employers will come to trust organizations that pre- 
pare trainees, and these organizations will become labor market intermediaries that over time 
become valuable to employers as a source of recruitment for qualified job applicants. Eventual- 
ly such providers also might be able to advocate on behalf of employees for better wages and 
working conditions. 

As displayed in Figure 1.1, the program includes the following key components: 

1 . Intensive screening of program applicants before enrollment — a practice 
not always common in employment programs offered to low-income indi- 
viduals — is intended to ensure that program providers select participants 
who can take advantage of the skills training for the sector and occupations. 

The expectation is that screening closely aligns with employer requirements 
for the targeted jobs. This was identified as one of the key elements of suc- 
cess in the SEIS. 

2. Sector-appropriate preemployment and career readiness services com- 
prise an orientation to the sector, career readiness training, individualized ca- 
reer coaching, and limited supportive services to sustain engagement and as- 
sist participants to complete their training and find employment. 
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The WorkAdvance Study 
Figure 1.1 

Program Logic Model 



NOTES: Program components are represented by boxes. Program outcomes are represented by circles. 

A limited number of participants were in a “placement- first” track and did not engage in occupational skills training. 








3. Sector-specific occupational skills training is intended to impart skills and 
lead to credentials that will substantially enhance workers’ employment op- 
portunities. Providers offer training tailored to current job openings in specif- 
ic sectors and occupations. In addition, providers are to adapt these offerings 
to changes in available jobs. This step did not apply to all enrollees in this 
evaluation. Early in the sample enrollment period approximately half the en- 
rollees at two of the four WorkAdvance providers were in a placement-first 
track in which they skipped the training phase (discussed further below). 

4. Sector-specific job development and placement services are intended to 
facilitate entry into positions for which the participants have been trained and 
for which there are thought to be genuine opportunities for continued skill 
development and career advancement. The providers’ job developers are ex- 
pected to maintain strong relationships with employers who hire individuals 
with the kinds of skills the program has imparted. 

5. Postemployment retention and advancement services are meant to assist 
participants beyond the placement stage. Providers are expected to maintain 
close contact with workers and employers to assess perfonnance, offer 
coaching to address any complicating life situations that might arise for 
workers, help identify next-step job opportunities and skills training to enable 
participants to move up career ladders over time, and assist with rapid 
reemployment if workers lose their jobs. By working closely with employers 
and understanding the career trajectories in their workplaces, providers 
should be able to help guide a participant’s career path. 

The Two-Track Model 

Although all the WorkAdvance providers eventually emphasized training before job 
placement, two of them (in northeast Ohio and Tulsa) implemented the program model with 
two separate tracks. This program design was suggested in the site selection process, based on 
previous workforce experience in New York City. One track emphasized gaining skills first 
through training (similar to most other sector-based programs), and the other sought to place 
people into jobs first; the goal for the two dual-track providers was to designate at least 50 per- 
cent of their participants to go into training first, while the remainder could be placed first. The 
placement-first track was intended to be a less expensive but still effective route to advancement 
by enabling enrollees to gain experience and sector-specific skills (such as through on-the-job 
training) without participating in formal training first. Another rationale for making placements 
right away was that it helped the providers offer and deliver a more immediate service to em- 
ployers while participants in the other track were still going through training and provider staff 
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members were building relationships with employers. However, at MDRC’s urging, both the 
providers eventually shifted mostly to the training-first approach, since providers were not go- 
ing to be able to reach the goal of enrolling 50 percent of participants in training if so many 
people continued to be placed first. Additionally, preliminary evidence suggested that place- 
ment first too often resulted in participants entering low-wage jobs that, in practice, did not lead 
to on-the-job acquisition of skills. 

The WorkAdvance Model in the Current Policy Context 

Why is a program like WorkAdvance needed? It is well documented that low-wage 
workers often experience unstable employment and modest wage increases. 1 1 WorkAdvance is 
designed to confront a number of obstacles that impede many low-wage workers (and programs 
serving them) from achieving labor market progression. These obstacles, and how WorkAd- 
vance attempts to address them, are summarized in Table 1.1. 

In theory, a program like WorkAdvance seems to have all the right elements for a suc- 
cessful employment program — so why is it even necessary to rigorously test it? In fact, there 
are several reasons why WorkAdvance might not be effective. First, participants, many of 
whom have limited resources, might not be able to complete the training. In addition, the train- 
ing might not meet employer needs, or the employers might not take the training as a serious 
signal of preparedness. There have been several instances of training programs that did not lead 
to advancement. 12 There are also some risks in the sectoral strategy. By focusing so much on a 
specialized segment of the labor market, providers and participants are particularly exposed 
when demand shifts. Finally, participants may get jobs in the sector, but low-wage workers of- 
ten face obstacles to keeping jobs. Life complications may interfere with them performing well 
on the job, which can lead to job loss. Therefore, while it seems obvious that employers need 
trained employees to work at jobs that require specialized skills, it is not obvious that programs 
like WorkAdvance will succeed in producing the right kind of labor supply for those jobs or that 
such programs will be an effective route to economic mobility for low-income people. 

The study is timely. There had been a number of sectoral programs even before the 
SEIS study, 13 but the promising findings of that study led to even greater interest in sectoral ini- 
tiatives, 14 including a major focus on such programs as part of the recent Workforce Innovation 


n Mishel, Bernstein, and Allegretto (2007); Schochet and Rangarajan (2004); Acs and Loprest (2004). 
12 Forone example, see Hendra et al. (2010). 

13 Project Quest in San Antonio (Osterman and Lautsch, 1996) and Focus: FIOPE in Detroit (Hollenbeck, 
2007) are notable examples. 

14 In a survey, the National Network of Sector Partners found that 47 percent of sectoral initiatives profiled 
were less than 5 years old (Mangat, 2010). Prominent initiatives founded between about 2005 and 2010 include. 
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The WorkAdvance Study 
Table 1.1 


Labor Market Obstacles and WorkAdvance Responses 


Obstacle 

WorkAdvance Element That Addresses Obstacle 

Human capital deficits 3 

• 

Occupational skills training 


• 

Career readiness training 


• 

Placement in jobs with skill acquisition po- 
tential 

Information and social capital deficits’ 5 

• 

Brokering with employers and intermediation 
role in job development/placement 


• 

Preemployment services to acclimate partici- 
pant to sector 

Costs of advancement 

• 

Free occupational skills training 


• 

Modest financial assistance to help partici- 
pants engage in program 

Lack of career opportunities 

• 

Advancement services/coaching for career 
navigation 

Employer hesitation to hire 

• 

Effective screening 


• 

Employer involvement in program design and 
adaptation 


• 

Career readiness services 

Misalignment of employment programs with 

• 

Focus of program staff on specific sectors 

labor market 


and the involvement of employers 


NOTES: “Human capital is generally defined as the skills, knowledge, and experience possessed 
by an individual or population, viewed in terms of their value or cost to an organization or 
country. 

b Social capital is generally defined as the networks of relationships among people who live and 
work in a particular society, enabling that society to function effectively. 


among many others, Skills2Compete Maryland, Pueblo Manufacturing Consortium in Colorado, and the South 
Central Pennsylvania Food Manufacturers’ Training Consortium. 




and Opportunity Act (WIOA). WIOA was passed with bipartisan support in July 2014, reau- 
thorizing the Workforce Investment Act (WIA) for the years 2015 to 2020. The bill specifically 
requires states to implement industry or sector partnerships and career pathways. 15 The Work- 
Advance findings, therefore, have direct relevance to WIOA, and the study’s lessons concerning 
postemployment and advancement services will have important implications as the new legisla- 
tion is implemented. In addition, WIOA includes a provision on service priorities that will en- 
sure that workforce efforts focus on a low-income population similar to the WorkAdvance sam- 
ple. Thus it is critical to understand these types of programs better, to confirm that they are ef- 
fective, to determine how they perfomi in different conditions and on a larger scale, and to 
measure how much they cost to operate. 

The design of WorkAdvance took into account several considerations, as discussed 

below. 


• WorkAdvance was specifically designed to be more than simply a job 
placement program because substantial evidence has accumulated and 
established that initial job placement alone is not sufficient to foster long- 
term advancement. 

The WorkAdvance model was built on several generations of rigorous random assign- 
ment evaluations of welfare-to-work and workforce programs that have sought to improve the 
employment outcomes of low-income people. The first strand of evidence comes from evalua- 
tions of programs designed to help people make the transition from welfare to work. 

The welfare-to-work studies yielded substantial knowledge about how to help low- 
income individuals prepare for and find jobs. The evaluations showed how numerous strategies, 
particularly those focused on getting participants into the labor market quickly, were quite effec- 
tive in increasing employment and reducing welfare. 16 However, many participants in the pro- 
grams that successfully boosted employment over a five-year follow-up period still ended up in 
unstable, low-paying jobs. Thus, the research suggested a need to focus on ways to effectively 
increase employment stability and wage progression. By the mid- to late 1990s, the federal gov- 
ernment and states focused squarely on the problem of employment retention and advancement. 
An initial multisite randomized controlled trial, the Post-Employment Services Demonstration 
(PESD), operated in the mid-1990s. It examined the effectiveness of offering certain services, 
such as counseling and support, frequent and flexible payments for work-related expenses, and 
other services, to newly employed welfare recipients. 17 The programs studied in the PESD, 


15 Workforce Innovation and Opportunity Act, Pub. L. No. 113-128, 128 Stat. 1425. 
16 Gueron and Rolston (2013). 

17 Rangarajan and Novak (1999). 
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however, had little effect on employment or earnings. This led to another, more ambitious eval- 
uation called the Employment Retention and Advancement project (ERA). 

ERA tried to go beyond the PESD models and tested a wide variety of strategies, which 
mostly focused on coaching, counseling, and case management. The models did not generally 
focus on skills training. The results of the ERA trials highlighted the difficulty of achieving up- 
ward mobility through simple strategic placement of people into jobs and generic on-the-job 
coaching alone. Of the 12 relevant programs studied in the ERA project, 18 only three were 
found to be effective at increasing earnings for participants. The nine unsuccessful programs 
offered guidance and advice after people found jobs (that is, postemployment), but little 
else. 19 These findings suggested that more needed to be done than simply helping participants 
navigate the labor market better. 20 One implication of the results was a refocus on skills training 
as a route to advancement, given the ERA evidence that counseling and coaching (as a supple- 
ment to work-first approaches) were insufficient to lead to advancement for most participants. 

As results from the PESD and ERA evaluations unfolded, some programs began to in- 
corporate more job training, acknowledging that some kind of vocational skill-building was 
needed in order to increase wages for low-wage workers. One initiative that attempted this was 
studied as part of the United Kingdom’s Employment Retention and Advancement project (UK 
ERA). The UK ERA results supported a long-standing lesson in the field of employment and 
training: Training does not work if it is not aligned with employer demand. The UK ERA pro- 
gram boosted training engagement, but labor market benefits attributable to Raining were not 
found, suggesting that there was a mismatch between the training undertaken and the labor 
market demand for individuals with that training. 21 The leading explanation for this result relat- 
ed to the program staffs capacity. The UK ERA advisory staff members functioned as em- 
ployment “generalists” — they offered participants general advice and guidance on adapting to 
work, encouraged them to consider seeking full-time work, helped them address issues of bal- 
ancing work and family life, advised them on seeking promotions and finding better jobs, and 
urged them to enroll in training courses in whatever areas interested them. But staff members 
did not have in-depth knowledge of particular occupations or industries, or expertise on the ca- 


18 The 12 programs exclude those targeted to “harder to employ” enrollees, such as individuals with sub- 
stance abuse issues. 

19 The programs were built upon a variety of hypotheses about what might be advantageous. For example, 
maintaining small caseloads; offering services at individuals’ workplaces; providing services through a collab- 
oration among welfare, WIA, and community college staff members; and continuing counseling relationships 
from pre- to post-job placement. None of these features produced sustained positive impacts on earnings, in 
and of themselves. (While the counseling and coaching produced a low yield on their own, researchers con- 
cluded that it was possible that these services could be very valuable when combined with other, more concrete 
services.) 

20 Hendra et al. (2010). 

21 Hendra et al. (2011). 
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reer ladders and training requirements for jobs in those areas. Nor did they steer participants 
assertively toward particular occupations known to offer real advancement opportunities. They 
were not positioned to connect participants who had trained in particular occupational areas 
with relevant employers who were hiring people with the new skills those participants had ac- 
quired. It is likely that these limitations undemiined the benefits of the extra participation in 
training that UK ERA produced. The findings point toward the value of providing career advice 
that is sector specific and more narrowly focused on opportunities available in the local labor 
market. 


A subsequent test of an approach with a more deliberate demand-driven focus occurred 
in the late 2000s, in the Work Advancement and Support Center (WASC) demonstration. The 
programs examined in WASC aimed to increase the incomes of low- wage workers by stabiliz- 
ing employment, building skills, increasing earnings, and easing access to work supports, such 
as child care subsidies and the Earned Income Tax Credit. One of the central hypotheses of 
WASC was that providing training through WIA One-Stop Career Centers would result in bet- 
ter aligmnent between training and work. Two of the WASC programs increased participation 
in education and training and also increased earnings in the third follow-up year. 22 In one pro- 
gram, these effects faded somewhat in the subsequent follow-up year; in the other, longer-term 
follow-up was not available. In both programs, the level of staff capacity to provide employer- 
informed advice was lower than anticipated. Still, because funding for training was mainly 
through WIA, there were conditions in place to try to ensure that training was in high-demand 
fields. In particular, in one of the programs, many of the training vouchers were used to pay for 
training in the rapidly growing health care field. These results suggested the promise of focusing 
training in high-demand areas, but doing so requires program staff members with deep 
knowledge of labor market niches: a central aspect of sector-based programs. 

• WorkAdvance was designed as a sectoral job training program that 
aligned skills training with job opportunities. 

One of the clearest findings from past research is that training works only when it corre- 
sponds to the labor needs of employers. 23 The sectoral approach is an effort to improve the 
aligmnent between training and employment opportunities. Beginning in the late 1980s, com- 
munity-based organizations across the United States pioneered workforce development pro- 
grams using a “sector strategies” approach. 24 

While early work by the Aspen Institute and CEO’s Workforce Innovations study of 
sector-focused programs found some encouraging evidence on the benefits of the sectoral ap- 


22 Miller, van Dok, Tessler, and Pennington (2012). 
23 Melendez (1996). 

24 Mangat (2007). 


11 


proach, 25 these studies were not as scientifically rigorous as randomized controlled trials; rather, 
the most powerful evidence to date comes from the aforementioned SEIS. 26 That study, which 
was a randomized controlled trial, found that sector-focused training programs for low-income 
workers and job seekers increased their earnings, employment, job stability, and access to bene- 
fits over a two-year period. Participants’ earnings over two years were $4,500 (or 18 percent) 
higher than earnings for the control group. 27 Earnings in the year after training were 29 percent 
higher than the control group average. Impacts from job training programs are usually much 
more modest, so these findings led to enthusiasm about sectoral programs. 

Key elements of the SEIS programs included the providers’ experience with sectoral 
programs; their strong relationships with local employers (“brokering”); provision of job readi- 
ness training in addition to occupational skills training; a stringent screening and intake process; 
and provision of individually tailored services. Although they aimed to place workers in “good” 
jobs — jobs that are higher-paying and more stable — the programs did not have an explicit 
advancement component. 

Building on the SEIS findings, it is also important to understand how well the sectoral 
strategy works with a wider range of providers. The three programs in the SEIS were carefully 
selected, and arguably represented unusually strong implementations of the sectoral approach. 
Notably, and in contrast with WorkAdvance, as this report explains, all three SEIS providers 
had been operating the program that was evaluated for at least three years before the study. Un- 
like in WorkAdvance, the providers were not asked to run a program model that had similar, 
specific components; rather, they were asked only to continue to run their programs “as is.” 
Therefore, one of the key contributions of WorkAdvance is to help shed light on whether a 
broader range of providers, less experienced with this particular strategy, can produce impacts 
similar to those measured in the SEIS. 

Finally, it is important to recognize that WorkAdvance sought to go beyond traditional 
sector programs. Traditional sector programs are built on the assumption that if providers find a 
sector in which there is room for advancement, workers will advance on their own given the 
right start. The explicit focus on advancement assistance as a means to upward mobility was an 
important enhancement introduced in the WorkAdvance model. Because the providers may not 
have been able to place participants directly into “middle-skill” jobs in all cases, the advance- 
ment coaching component of the model was hypothesized to help participants make the transi- 
tion from an entry-level job to a middle-skill job after they gained some additional experience. 


25 Zandniapour and Conway (2002); Henderson, MacAllum, and Karakus (2010). 
26 Maguire et al. (2010). 

27 Maguire et al. (2010). 
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Selection of WorkAdvance Program Providers 

Under the auspices of the federal SIF grant, the WorkAdvance managing consortium 
set out to competitively select local providers in three geographic areas — New York City; 
northeast Ohio (Greater Cleveland and Youngstown); and Tulsa, Oklahoma — rather than to 
conduct a broader national search . 28 A principal consideration in selecting the geographic areas 
was demonstrated local interest, as represented by a commitment to help raise the match fund- 
ing needed to satisfy the SIF requirements. A second important factor was that there be a diver- 
sity of local economies and industry sectors, so that the WorkAdvance demonstration could 
draw on a range of local conditions to inform potential replication and expansion of similar pro- 
grams. 


The four providers that were selected for WorkAdvance had a range of experiences and 
backgrounds. Ultimately, this benefited the study, as it provided the opportunity to leam, among 
other things, whether an array of providers — including some that were less mature in their de- 
livery of sectoral programs than those included in the SEIS or that had no sectoral experience at 
all — could successfully implement WorkAdvance. 

The WorkAdvance providers were selected by CEO and the Mayor’s Fund to Advance 
New York City through a competitive process conducted within each of the identified geo- 
graphic areas, with input from MDRC and local stakeholders from each region, including repre- 
sentatives of government and philanthropy. A primary factor in selection decisions was whether 
a provider could demonstrate that it was currently, or had the capability to be, firmly grounded 
in a targeted sector; this included in-depth knowledge of and strong relationships with employ- 
ers who provided letters of support. Applicants had to demonstrate current or potential capacity 
to operate at the intended scale, to carry out an advancement-focused approach, and to work 
with a range of lower-income individuals — rather than only those who would be easiest to 
place in jobs. Additional selection criteria included overall organizational capabilities (including 
appropriate fiscal and data management capacity and the ability to comply with federal funding 
requirements), clear commitment to the program model, and a willingness and ability to partici- 
pate in a random assignment study and to help raise matching local funds. All the selected pro- 
viders demonstrated the commitment of the agency’s leadership to the requirements of the 
WorkAdvance demonstration. 


28 The WorkAdvance managing consortium members are as follows: The Mayor’s Fund to Advance New 
York City was the grantee of the SIF; the New York City Center for Economic Opportunity (CEO) managed 
the contracts with the providers (called “subgrantees”) and was accountable to the funders for program imple- 
mentation and progress, and for operating a Learning Network for the project; and MDRC provided technical 
assistance to the providers in the implementation of WorkAdvance, monitored their performance (along with 
CEO), and led the evaluation. 
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CEO and the Mayor’s Fund did not specify the targeted sectors in each geographic area; 
instead, providers proposed and had to justify the sector and range of occupations, based on 
their experience, local labor market infonnation, and the advancement potential of the targeted 
jobs. 


The four organizations selected to operate WorkAdvance are as follows: 

• Per Scholas in the Bronx, New York, which focused on the infonnation 
technology (IT) sector 

• St. Nicks Alliance in Brooklyn, New York, which focused on enviromnental 
remediation and related occupations 

• Madison Strategies Group in Tulsa, Oklahoma, which focused on the trans- 
portation and manufacturing sectors 

• Towards Employment in northeast Ohio, which focused on the health care 
and manufacturing sectors 

Table 1.2 provides some background on the four WorkAdvance providers. Chapter 2 provides 
more details, particularly with regard to the organizations’ levels of experience in operating sec- 
toral training programs. All four programs continue to provide services similar to those offered 
under WorkAdvance. 


The WorkAdvance Evaluation Design 

Random Assignment 

The WorkAdvance programs were studied using a random assignment design, a meth- 
odology that allows practitioners and policymakers to have a high degree of confidence in the 
results. Random assignment designs are known as the “gold standard” because the randomiza- 
tion process removes potential sources of bias, creating two groups that are expected to be 
equivalent at baseline. The WorkAdvance study also has high external validity — or the extent 
to which the results can be expected to apply to other settings — because the intervention was 
tested in four diverse sites and across several sectors. For these reasons this study provides the 
highest level of evidence, according to SIF criteria. 29 

From June 2011 through June 2013, individuals who met the WorkAdvance and sector- 
specific eligibility criteria were assigned at random to the WorkAdvance group or to the control 


29 The SIF defines an example of the highest level of evidence as a large, well-designed and well- 
implemented, randomized controlled multisite trial that supports the effectiveness of the practice, strategy, or 
program. This study meets those criteria. 
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The WorkAdvance Study 
Table 1.2 


Institutional Structure of Providers 



Per Schoias 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Location 

Bronx, New York 

Brooklyn, New York 

Tulsa, Oklahoma 

Cleveland, Ohio 

Year founded 

1995 

1975 

2011 

1976 

Sector(s) 

Information technology 

Environmental remediation 

Transportation and manufacturing 

Health care and manufacturing 

Key 

organizational 

characteristics 

Provides in-house information 
technology training and job 
placement services. 

Large social service agency 
offering housing, health care, 
workforce and economic 
development, and youth and 
education services. WorkAdvance 
operates within a broader 
workforce development unit at St. 
Nicks Alliance. 

New workforce nonprofit 
operating only WorkAdvance, 
built as the offshoot of a for-profit 
workforce agency operating in 
New York City. 

Employment-focused organization 
offering career readiness training, 
supportive services, access to 
occupational skills training, and 
employer partnerships. 

Approach 

Training-first for all enrollees 

Training- first for all enrollees 

Mixed enrollment to training- and 
placement-first 

Mixed enrollment to training- and 
placement-first 


SOURCES: Documentation provided by providers. 























group. 30 Members of the WorkAdvance group were offered WorkAdvance services, while those 
in the control group were not eligible for WorkAdvance services but were eligible for other ser- 
vices available in the community. 31 The first six months of random assignment consisted of a 
pilot phase, designed to provide the organizations with time to work out implementation issues 
and to learn to perform research procedures (such as collecting baseline data, explaining in- 
formed consent, and conducting random assignment). A pilot assessment was conducted to de- 
termine how well the program was implemented during the pilot phase. The assessment con- 
cluded that the program was sufficiently well implemented to enable the pilot sample to be in- 
corporated in the research sample. 

Both the WorkAdvance group and the control group were tracked over time. The ran- 
dom assignment process ensured that, when individuals entered the study, there were no sys- 
tematic differences in sample members’ characteristics, measured or unmeasured, between the 
two research groups. Thus, any differences between them that emerge after random assignment 
can be attributed to the location’s WorkAdvance program. 

The Role of Technical Assistance 

WorkAdvance was a multicomponent program that required all providers to go beyond 
their normal operations. All providers received extensive technical assistance during the study 
period, to ensure that research procedures were being followed and to ensure that the providers 
were delivering the strongest program possible, according to the way it was designed. MDRC 
and several consultants provided technical assistance. 

Components of the Study and Data Sources 

This section lays out the components and the key questions addressed in each analysis 
and the key data sources. Table 1.3 provides more detail on the sources. 

The implementation analysis assesses how each provider put the key components of the 
WorkAdvance model into operation. The analysis also examines what it takes to operate the 


30 Random assignment began at Per Scholas first. As Figure 1.2 shows, random assignment began later in 
2011 at the other three sites. The programs continued to provide sectoral training and employment services 
after the random assignment period concluded. 

31 MDRC conducted random assignment using a secure, online web-based random assignment system that 
has been used in many MDRC studies. The random assignment algorithm has been extensively checked and 
random assignment was monitored by analyzing the percentage of assignments to both groups. Extensive sta- 
tistical tests were conducted to ensure that random assignment produced two similar groups; that is, that base- 
line characteristics did not vary systematically across the research groups. These procedures were the same at 
each site and are described in further detail in Appendix A. See Appendix Table C.2 for more information on 
intake and the point of random assignment. 
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The WorkAdvance Study 
Table 1.3 
Data Sources 

Data Source Description 

Baseline data Baseline data were collected by MDRC at the time of random assignment using an online 
module. Data include information on demographic characteristics, prior employment, and 
various other characteristics. This information is used to describe the study samples, to create 
subgroups, and to refine statistical estimates. 

Field research Field research is the main data source for the analysis of program features and implementation. 

The field research includes MDRC’s observations of and interviews conducted with the 
providers’ staff members, observations of participants, review of program materials (for 
example, curricula for career readiness), and operational experience. Further information 
collected from five focus groups of between 9 and 12 WorkAdvance group members each and 
individual interviews conducted with 20 WorkAdvance group members at two points in their 
program tenure is used to provide examples of how WorkAdvance was implemented across 
the providers. Two special analyses were conducted: a “funnel analysis,” which provides data 
on recruitment and the flow of applicants through the study intake process, and an analysis that 
documented the frequency and content of advancement coaching sessions. (These analyses are 
described in more detail in Appendix A and Chapter 2, respectively.) 

Program Program tracking data are used as a source of information for WorkAdvance group activities, 

tracking data supportive services receipt, education and training funds (where applicable), operating costs 

(administrative and staff), and job placement activities. These data are used to estimate costs of the 
program and to describe the program services (as a supplement to the field research data). These data 
were provided over the course of the study by the providers using a template created by MDRC. 
These data were collected only for the WorkAdvance group. 

Administrative Effects on employment and earnings are computed using unemployment insurance (UI) wage 

records records data provided by the state employment agencies during the course of the study. It is 

estimated that Ul data cover approximately 90 percent of all jobs. 3 All unreported employment is 
missing. In addition, UI benefits data, where available, are used to measure whether increases in 
employment via WorkAdvance translate into reductions in the take-up of UI benefits. The same data 
were collected for both research groups. 

Survey data Information about sample members’ experiences with program operations and services, as 
well as their employment outcomes, was collected through a survey administered to 
WorkAdvance and control group members approximately 1 8 to 24 months after their random 
assignment. The survey achieved a response rate of 81 percent. b The survey focused on 
contact, message and help received (for example, for job placement), participation in a job 
search and education and training, and characteristics of employment, such as the number of 
months employed, hours worked, hourly and monthly wages, and industry and occupation. 

The survey also captured various secondary outcomes. 

NOTES: a Kornfeld and Bloom (1999). A WorkAdvance-specific estimate of UI coverage is presented in 
Appendix A. 

b A survey response analysis is shown in Appendix A. While there are some minor differences between survey 
respondents and nonrespondents, there are no major concerns about the representativeness of the survey data. 
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WorkAdvance model well, what organizational and other factors influenced operations, and 
how the programs evolved over time. The analysis relies on data collected through a combina- 
tion of site visits, interviews, focus groups, program records, and study-specific site data collec- 
tion efforts. 

The participation analysis measures the differences in services received by the Work- 
Advance group and the control group, using survey data. Services measured include not only 
those provided by WorkAdvance, but also other services generally available in the community. 
By establishing the service contrast that WorkAdvance produced, the analysis provides critical 
context for interpreting the results from the economic impact analysis. 

The cost analysis estimates the average per person cost of operating the WorkAdvance 
program. It first estimates the gross cost per person by looking at the full costs of services deliv- 
ered to the WorkAdvance group divided by the number of WorkAdvance group members. It 
then estimates WorkAdvance’s net cost per person by subtracting the average costs of the ser- 
vices that the control group received from the average gross cost of providing services to the 
WorkAdvance group. 

The economic impact analysis focuses on measuring the extent to which WorkAdvance 
affected labor market outcomes, such as employment, employment retention, earnings, wage 
rates, hours worked, employer-provided benefits, and advancement, beyond the levels attained 
by control group members over the follow-up period. These outcomes are measured by Year 2 
survey data and (in the case of earnings and employment) unemployment insurance (UI) wage 
data. The analysis also examines the effects the program had on secondary measures such as life 
satisfaction and material hardship. 

Key Samples and Follow-Up Periods 

Given the differences in starting points and organizational emphases across the four 
providers involved in WorkAdvance, a decision was made early in the project to give priority to 
analyzing impacts by site. 32 The main exception is for the subgroup analysis, which focuses on 
the pooled sample. This exception is necessary because sample sizes for subgroups are not large 
enough at the site level to enable precise estimates. 


32 In the Subgrantee Evaluation Plan prepared for WorkAdvance as part of the SIF, the original plan was to 
conduct the analysis by sector. Because most sites focused on a unique sector, the only difference between 
analyzing impacts at the site level and at the sector level is in Ohio. Because one of the original Ohio programs 
(Compass) was excluded (see Tessler et al., 2014), there was insufficient sample size to analyze the Ohio sec- 
tors separately. Appendix A discusses this in more detail. 
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Figure 1.2 shows the sample size for each site. Sample sizes range from 479 at St. 
Nicks Alliance up to 698 at Towards Employment. The pooled sample size is 2,564. 33 Survey 
sample sizes are roughly 80 percent of these totals. The overall study sample is more than twice 
as large as the sample enrolled in the SEIS, and the average WorkAdvance site sample is more 
than twice as large as the average SEIS site sample. Therefore, in addition to enlisting a more 
diverse set of providers (in terms of their experience operating sectoral training programs), the 
WorkAdvance study also built on the SEIS by providing estimates of how these programs 
would perfomi on a larger scale. 

The full sample has approximately two years of common follow-up from both adminis- 
trative records and the follow-up survey. The average survey respondent was interviewed in 
Month 22. 34 Two and one-quarter years of UI wage data are available for the full sample. 

It is well established in the job training literature that at least two years of follow-up is 
needed to measure the impacts of training programs, because it takes time for participants to 
complete their training, find a job, and advance in the job. For example, in the SEIS, positive 
impacts did not emerge until late in Year 1; impacts were negative early in Year 1 due to the 
opportunity cost of time spent in training instead of work. In other studies, impacts have taken 
as long as three years to emerge. Therefore, the amount of follow-up analyzed in this report may 
not be enough to capture WorkAdvance’s long-term impacts (particularly the impacts of ad- 
vancement services). As previously noted, plans are in place to collect and analyze longer-term 
follow-up data for the study sample. 

Characteristics of the WorkAdvance Sample 

WorkAdvance targeted unemployed and low-wage working adults with a monthly fam- 
ily income below 200 percent of the federal poverty level. As a proxy for the targeted low-wage 


33 The target sample size was originally 3,850. Recruiting a sample that large within the confines of the SIF 
grant period proved infeasible. A statistical power analysis, discussed further in Appendix A, verified that the 
study is more than adequately powered at the sample size that was achieved. All but 48 individuals recruited 
for the study are included in the research sample. Attrition from the study was minimal: A total of 20 individu- 
als (0.8 percent) withdrew from the study (10 from the Per Scholas site, 3 from the St. Nicks Alliance site, 2 
from the Madison Strategies Group site, and 5 from the Towards Employment site). Two of the withdrawals 
were WorkAdvance group members, and 18 were control group members. Apart from these withdrawals, par- 
ticipants were excluded from the impact analysis for a few reasons: 22 were being served under another fund- 
ing stream, 1 was missing a signed consent form, and 5 were enrolled before the official start of random as- 
signment. Of the 2,564 individuals in the research sample, about half (1,293) were assigned to the program 
group and were eligible to receive WorkAdvance program services. The vast majority (95 percent) of program 
group members received at least one program service within the first 18 months of follow-up. (See Chapter 3 
for more details on service take-up.) 

34 According to the original Subgrantee Evaluation Plan, the survey was to be launched 1 8 months after 
baseline, but it was moved to the two-year point to extend the follow-up period. 
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The WorkAdvance Study 
Figure 1.2 

Random Assignment Period and Report Sample Sizes, by Site, Analysis Sample, and Research Group 


Analysis sample 

Per Schoias 

Report sample 

Year 2 Survey respondent sample 

St. Nicks Alliance 

Report sample 

Year 2 Survey respondent sample 

Madison Strategies Group 

Report sample 

Year 2 Survey respondent sample 

Towards Employment 

Report sample 

Y ear 2 Survey respondent sample 


Random Assignment Period 


2011 

2012 

2013 

Sample Size 


J J A S 0 N D 

JFMAMJ JASOND 

J F M A M J 

WA Group C Group 

Total 





349 341 690 

287 265 552 


242 237 479 

205 179 384 

353 344 697 

297 263 560 

349 349 698 

286 276 562 





SOURCE: MDRC calculations from WorkAdvance baseline information form. 


NOTE: WA group = WorkAdvance (program) group; C group = control group. 




worker population, adults who were working at the time that they entered the study were also 
required to be earning less than $15 per hour. 35 

Table 1.4 presents selected characteristics at baseline of the WorkAdvance research 
sample by provider and overall. 36 The variation in baseline characteristics that is seen across 
providers is likely to reflect provider-specific eligibility criteria, the sectors selected, and the city 
in which the provider is located. Various tests for baseline equivalence between the WorkAd- 
vance group and the control group are presented and discussed in Appendix A. As expected (by 
virtue of the random assignment research design) there are no meaningful or systematic differ- 
ences between the research groups. 

The average WorkAdvance sample member is 34 years old, black/Alfican-American, 
and single, with some variation seen across the sectors. Notably, the Per Scholas sample mem- 
bers are younger and much less likely to have children. The majority of WorkAdvance sample 
members are male, with the exception of the health care sector at Towards Employment (not 
shown separately), in which more than 92 percent of sample members are female. Almost all 
sample members had at least a high school diploma or General Educational Development 
(GED) certificate, and over half the sample had at least some college education — a much high- 
er rate than seen in the SEIS. Individuals at the St. Nicks Alliance site had lower levels of edu- 
cation than at the other sites. 

Early in the follow-up period in particular, the sample composition seemed heavily af- 
fected by the Great Recession and the accompanying labor market displacement that took place 
in 2008-2009. These events pushed many previously stably employed individuals into unem- 
ployment and led to declines in the economic circumstances of many formerly “middle-class” 
individuals. Almost all sample members had previous work experience, but only one in five 
were working at the time that they entered the study. In addition, more than 36 percent of sam- 
ple members had been unemployed for at least seven months before study entry. Overall, 15 
percent of sample members had previous work experience in their targeted industry; this rate 
varied from 2 percent at St. Nicks Alliance to 31 percent at Towards Employment, which fo- 
cused partly on health care, an industry which employs a large number of workers and which 
has low barriers to entry-level jobs, such as home health aides and nursing assistants. Despite 
fairly high levels of education and work experience, many sample members still faced substan- 
tial barriers to employment, and many were receiving public benefits. One-quarter of the overall 


35 In addition to study eligibility criteria, some of the providers also imposed eligibility criteria related to a 
given sector and the occupations for which their training programs prepared participants. See Appendix Table 
C.2 for details. 

36 Additional sample characteristics are presented in Table B. 1. 
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The WorkAdvance Study 
Table 1.4 


Selected Characteristics of Research Sample Members at Baseline and Comparison 
to National Low-Wage Workers, Cross-Site 


Characteristic 

PS 

SNA 

MSG 

TE 

WA 

Overall 

United 

States 

WorkAdvance group (%) 

50.6 

50.5 

50.6 

50.0 

50.4 


Demographic characteristics 

Female (%) 

13.2 

14.5 

15.9 

58.6 

26.6 

57.0 

Average age (years) 

31 

35 

35 

35 

34 

38 

Age (%) 

18-24 years 

31.4 

16.3 

22.4 

22.9 

23.8 


25-34 years 

39.6 

37.2 

33.9 

29.2 

34.8 


35-44 years 

16.1 

25.7 

20.4 

21.2 

20.4 


45-59 years 

12.8 

19.8 

22.2 

24.9 

20.0 


60 years or more 

0.1 

1.0 

1.1 

1.7 

1.0 


Race/ethnicity (%) 

Latino/Hispanic 

36.0 

22.8 

6.1 

4.6 

16.8 

27.6 

White 

5.2 

6.9 

39.2 

18.4 

18.4 

45.7 

Black/African- American 

44.5 

62.7 

28.4 

70.8 

50.6 

18.9 

Other race 3 

14.3 

7.5 

26.4 

6.2 

14.1 

7.8 

Born in United States (%) 

71.9 

76.9 

95.4 

97.9 

86.3 

75.2 

Family status (%) 

Marital status 

Single, never married 

76.7 

70.9 

50.6 

72.7 

67.4 

46.8 

Married and living with spouse 

11.9 

14.3 

22.8 

9.6 

14.7 

31.0 

Married but living apart from spouse 

6.1 

4.0 

6.1 

6.2 

5.7 

1.9 

Legally separated, divorced, or widowed 

5.4 

10.9 

20.6 

11.5 

12.2 

20.3 

Parent of one or more children 

26.2 

45.3 

51.8 

51.8 

43.7 

44.3 

Education level (%) 

Highest level of education attainment 

Less than GED certificate or high school diploma 

0.1 

11.9 

6.0 

5.9 

5.5 

27.3 

GED certificate/high school diploma 

37.1 

44.5 

35.7 

37.1 

38.1 

35.9 

Some college 

32.5 

26.5 

48.1 

47.0 

39.5 

20.3 

Associate's degree/2-year college 

9.9 

7.5 

5.7 

4.9 

6.9 

6.2 

4-year college degree or more 

20.4 

9.6 

4.4 

5.2 

9.9 

10.3 

Already has a license/certificate in targeted industry 

3.6 

1.9 

13.5 

24.9 

11.8 


Employment status 

Ever employed (%) 

96.4 

97.9 

99.3 

97.3 

97.7 



(continued) 
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Table 1.4 (continued) 


WA United 


Characteristic 

PS 

SNA 

MSG 

TE 

Overall 

States 

Number of months of current unemployment spell (%) 

Never employed 

3.6 

2.1 

0.7 

2.7 

2.3 

3.4 

Currently employed 

13.0 

10.5 

26.7 

26.6 

20.0 

45.9 

12 months or less 

56.1 

55.9 

58.0 

44.0 

53.3 

10.5 

6 months or less 

41.3 

41.2 

49.9 

32.7 

41.3 


7-12 months 

14.8 

14.7 

8.0 

11.3 

12.0 


More than 12 months 

27.2 

31.5 

14.6 

26.6 

24.4 

40.1 

Number of months in current or most recent job (%) 

12 months or less b 

55.5 

54.0 

65.9 

58.3 

58.8 


More than 12 months 

44.5 

46.0 

34.1 

41.7 

41.2 


Is or has been employed in targeted industry (%) 

7.9 

2.1 

14.7 

30.6 

14.9 


Average hourly wage at current or most recent job c ($) 

11.91 

13.00 

10.32 

9.88 

11.12 


Among those currently working 

10.05 

10.84 

9.54 

9.46 

9.72 

9.77 

Worked full time (35 or more hours per week) d (%) 

58.6 

65.1 

71.4 

61.1 

64.0 


Among those currently working 

12.2 

28.0 

46.8 

43.5 

37.7 

29.8 

Average weekly wage at current or most recent job L ($) 

408 

454 

400 

342 

396 


Circumstances that mav affect job chanee or retention 1%) 

Previously convicted of a crime 

10.4 

19.9 

39.7 

25.4 

24.2 


Previously incarcerated 

6.2 

18.2 

34.2 

12.0 

17.6 


Income and medical coveraee 

Average monthly family income ($) 

647 

695 

804 

600 

686 


Income sources (%) 

Food stamps/SNAP 

17.4 

41.9 

34.5 

55.2 

36.9 


Welfare/TANF 

6.2 

13.7 

0.7 

3.9 

5.5 


Unemployment insurance benefits 

24.3 

24.6 

7.5 

10.5 

16.0 


Covered by health insurance plan (%) 

54.3 

54.3 

28.3 

48.1 

45.6 


Sample size 

690 

479 

697 

698 

2,564 

8,946 


SOURCES: MDRC calculations from the WorkAdvance (WA) baseline information form and March 2012 Current 
Population Survey. 

NOTES: PS = Per Scholas; SNA = St. Nicks Alliance; MSG = Madison Strategies Group; TE = Towards 
Employment; WA = WorkAdvance group; GED = General Educational Development. 

Sample sizes may vary because of missing values. 

Italics indicate the metric is not among the full sample shown in the table. 

Low-wage workers for the U.S. sample are defined as individuals at least 18 years old, with a family income of 
less than 200 percent of the federal poverty level, not retired, and earning less than $15 per hour if currently 
employed. The definitions of "family" differed across data sources. 

a "Otherrace" includes sample members who identify as non-Hispanic and listed "Asian," "American Indian," or 
"other" as their race, including sample members who answered as "multiracial." 
b Pc r centage includes sample members who have never been employed. 
c Wages for sample members who have never been employed are counted as $0. 

d "Worked frill time" does not include the 2.3 percent of sample members who have never been employed. 
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sample had a previous criminal conviction, and even higher rates were seen within the transpor- 
tation and manufacturing sectors. At study entry, less than half the sample members were cov- 
ered by health insurance, 6 percent were receiving Temporary Assistance for Needy Families 
(welfare), 16 percent were receiving unemployment insurance benefits, and over one-third were 
receiving food stamps. 

Table 1.4 also presents a column showing selected baseline characteristics for a national 
sample of low-wage workers. While WorkAdvance targeted all unemployed and low-wage 
workers, the sample ultimately enrolled in the study reflects the providers’ choice of sectors and 
recruitment methods, and the study and provider screening criteria. There are a few notable dif- 
ferences between the WorkAdvance sample and the national low-wage worker sample. Over 
half of the national sample is female, compared with only 27 percent of the WorkAdvance sam- 
ple. This is probably due in part to the sectors targeted by the WorkAdvance providers; all of 
the sectors, with the exception of health care, tend to be male dominated. The WorkAdvance 
sample had relatively higher levels of education: Over one-quarter of the national low-wage 
worker sample does not have a GED or high school diploma, compared with only 6 percent of 
the WorkAdvance sample. Few individuals in either sample lack work experience, but the na- 
tional sample has higher proportions of individuals who are currently employed and individuals 
who have been out of work for over a year (not shown). 

Economic Conditions During the Study Period 

WorkAdvance began operations in June 2011, random assignment concluded in June 
2013, and this report covers outcomes through September 2015. The WorkAdvance study coin- 
cided with a time of fairly steady economic growth despite occurring during the slow recovery 
period following the Great Recession — a period when even relatively experienced and skilled 
workers struggled to find work. The early part of the recovery was notable for its lack of job 
creation and earnings growth. This was compounded by the fact that the period up to 2007 was 
sometimes called the “jobless recovery.” Thus, low-wage workers confronted an extended peri- 
od of labor market stagnation. 37 Recent studies indicate that employers have responded to this 
increased supply of unemployed workers by being especially selective about whom they hire, 
particularly in relation to recent work experience. Those who have been out of the labor market 
for six months or longer are much less likely to receive calls for job interviews, even when ap- 
plicants have extensive relevant experience. 38 Despite these challenging economic conditions, 
during the study period unemployment fell by 2.7 percentage points nationally — from 8.9 per- 
cent in 2011 to 5.3 percent in 2015. There was some important economic variation by site loca- 
tion, as shown in Appendix B. Notably, the economy was especially strong during the study 


37 Kolesnikova and Liu (2011). 

38 Ghayad (2013); Kroft, Lange, and Notowidigdo (2013). 
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period in Tulsa due to a local economic boom related to the natural gas and oil industries. While 
the economy in New York City was improving throughout the evaluation period, the environ- 
mental remediation industry (which the St. Nicks Alliance program focused on) was quite slack 
early in the follow-up period. In Ohio, the economy was fairly steady. 


The Organization of This Report 

The report is organized according to five primary questions outlined in the WorkAdvance Sub- 
grantee Evaluation Plan (SEP) prepared as part of the SIF: 39 

• How did the different providers implement the program model, and what ad- 
justments did they make over time? (Chapter 2) 

• To what extent did WorkAdvance increase engagement in employment and 
training services, compared with what would have happened in the absence 
of the program (represented by the control group)? (Chapter 3) 

• What does it cost, on average, to operate the WorkAdvance program? (Chap- 
ter 4) 

• To what extent did WorkAdvance improve primary economic outcomes 
(employment and earnings) and secondary outcomes (such as income and 
life satisfaction) over the follow-up period? (Chapter 5) 

• What helps explain the economic impact results — the targeted sector? The 
characteristics of the study participants? Did these impacts change over time 
as providers matured? (Chapter 6) 

• The conclusion (Chapter 7) describes the WorkAdvance results within the 
broader framework of other employment and training studies. 


39 Deviations from the original questions included in the SEP were minor. As mentioned above, one shift 
was a focus on impacts by sector instead of by site (which affected only the Ohio site). In addition, the SEP 
specified that the evaluation would attempt to measure impacts on skills. This effort was abandoned, except to 
the extent that impacts on credentials are a proxy for impacts on skills. Finally, the WorkAdvance SEP raised 
the possibility of conducting a full benefit-cost analysis. Current plans are to conduct this analysis after five 
years of follow-up data have been obtained, to obtain a more reliable estimate of benefits. 


25 




Chapter 2 

Implementation of the WorkAdvance Program 


Chapter Highlights 

• The WorkAdvance intake screening process (which took place before ran- 
dom assignment occurred) was a key part of the program. 

• Engagement of program enrollees in program services, particularly occupa- 
tional skills training, was high across all four providers. 

• Postemployment advancement coaching was the last program component to 
be implemented, but it was in place during the latter half of the service deliv- 
ery period. 


Introduction 

The degree to which a program like WorkAdvance is implemented as intended can be influ- 
enced by a wide variety of factors: the population served, the organizational history and ap- 
proach of the provider, staff capacity, local economic conditions, and the strengths or limitations 
of the chosen sector. The ways in which WorkAdvance was implemented, and how implemen- 
tation differed across providers, is the focus of this chapter. An important point is that all major 
WorkAdvance program components were eventually implemented at all four providers. 

In addition to updating the implementation findings presented in the previous 
port, 1 the chapter describes the implementation of the program components that were offered to 
participants for up to two years after enrollment: preemployment and career readiness services, 
occupational skills training, job development and placement, and postemployment retention and 
advancement services. The chapter particularly focuses on retention and advancement services, 
which were the last set of WorkAdvance services to be implemented at all providers. It also up- 
dates participation statistics relative to the previous report, providing a total of 18 months of 
follow-up for the full program group sample — that is, for the 1 ,293 unemployed and low-wage 
working adults who were randomly assigned to the WorkAdvance group from June 2011 (the 
start of the WorkAdvance program) to June 20 13. 2 


'Tessler et al. (2014). 

‘Another 1,271 individuals were randomly assigned to the control group during this period, resulting in a 
total, cross-site sample size of 2, 564 individuals. 
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Analyses in the chapter draw on a variety of data sources. Qualitative data come from 
interviews with staff, participants, and employers; observations of participant-staff advancement 
coaching meetings; reviews of enrollee case files; and focus groups with program participants. 
Quantitative data on the recruitment and the flow of participants through the screening and en- 
rollment process come from the providers’ management infomiation systems (MIS), and data 
that documented the frequency and content of advancement coaching sessions come from a 
special study. The qualitative data cover program operations through June 2014; the quantitative 
data used in this chapter go through December 2014. 

Both qualitative and quantitative data were analyzed to understand several program im- 
plementation dimensions: 

• What organizational and other factors influenced the operations of WorkAd- 
vance at each site? 

• How closely did program operations hew to the original WorkAdvance pro- 
gram design, and how did they evolve over time? 

• To what extent did participants engage in the various WorkAdvance program 
components? 

• How well did program staff members deliver services, particularly post- 
employment services, to identify and support advancement opportunities? 

• To what extent did participants complete training and obtain credentials? 

• What were participants’ views of the program? 

In the analysis, qualitative data were coded into themes and trends and then compiled 
across codes and analyzed by theme. Quantitative data were generally analyzed for set follow- 
up periods relative to each program enrollee’s random assignment date. See Appendix A for 
more details on data collection and analysis. 


Key Findings 

• The WorkAdvance intake and screening process, which occurred before 
random assignment, resulted in many interested individuals being iden- 
tified as not academically qualified or self-selecting out of the process. 
This process increased the recruitment burden on the sites but may have 
contributed to the high program activity participation and completion 
rates. 
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Only about 20 percent of the individuals who initially expressed interest in WorkAd- 
vance were both still interested in and qualified for enrollment by the end of the program intake 
process, before random assignment was conducted. The process included several steps, and in- 
terested individuals were asked to report to the provider on multiple occasions. This thorough 
process was in itself a screen for individuals’ motivation to participate in the program. Most 
people who did not eventually enroll in the program either withdrew of their own accord or 
failed to achieve the required score on the academic assessments, which reflected sector and 
occupation standards. While the rigorous intake process was a burden on the providers, it prob- 
ably contributed to the high participation and completion rates discussed below. 

• Translating the WorkAdvance model into a set of concrete, implement- 
ed services took time — more than a year for some components and 
providers — and all four providers required a substantial amount of 
technical assistance and related support, although some required more 
time and assistance than others. 

The program model was designed specifically for the WorkAdvance demonstration, 
and each provider had little or no experience with at least one of the WorkAdvance compo- 
nents, especially the postemployment emphasis on career advancement rather than just job re- 
tention. The model also demanded that providers have a strong capability to work with employ- 
ers, training partners, and program participants. In addition, preparing for and carrying out 
random assignment was labor intensive and impinged on program operations at times. The pro- 
viders’ varying degrees of experience in each of these areas influenced their implementation of 
the program in different ways, as described in the chapter. 

• All four providers eventually delivered services across all the WorkAd- 
vance model components, with postemployment services the last to be 
fully developed and implemented. The engagement of program enrollees 
in key program components was high across all four providers. 

As illustrated in Table 2.1, which summarizes key findings on participants’ engagement 
in program services, close to 95 percent of WorkAdvance group members engaged in career 
readiness activities. Within 18 months of enrollment, around 76 percent started occupational 
skills training, and about 63 percent of all WorkAdvance group members completed it. In most 
cases, completion of occupational skills training led to the earning of a nationally or locally rec- 
ognized credential (or both) — a critical first step toward getting a job in a sector. Across all 
providers, a high proportion of all WorkAdvance enrollees — 55 percent — earned such licens- 
es or certificates. 
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The WorkAdvance Study 
Table 2.1 


Summary of Indicators of Participation in Program Group Activities Within 
18 Months of Random Assignment, Cross-Site 


Participation in program activity since RA (%) 

Per 

Scholas 

St. Nicks 

Alliance 

Madison Towards 

Strategies Group Employment 

Overall 

Ever participated in any career readiness activity' 1 

95.7 

82.6 

98.3 

98.3 

94.7 

Ever started skills training 

95.7 

82.6 

62.0 

65.3 

75.9 

Ever completed skills training 

78.5 

78.9 

48.7 

50.1 

62.8 

Ever obtained a credential 13 in targeted sector 

72.5 

73.6 

48.7 

31. 8 C 

55.2 

Sample size 

349 

242 

353 

349 

1,293 


SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 
Strategies Group (MSG), and Towards Employment (TE). 


NOTES: RA = random assignment. 

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. 
a The first program activity at PS and SNA is skills training, which is offered in combination with career 
readiness training and includes help with developing a career plan, resumes, and interview preparation. At MSG, 
the first progam activity is career readiness training. At TE, the first program activity is an initial assessment 
wherein career goals and barriers to employment are discussed. 

b Credentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 
reporting of locally recognized credentials obtained in the targeted sector. 

c TE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 
obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 
having obtained a credential. 


Postemployment advancement coaching did not occur in earnest until after many of the 
other WorkAdvance program components were put in place, and for many providers not until 
sample recruitment was complete — roughly two years after WorkAdvance started. Still, in the 
later part of the follow-up period it appears that this service was offered more systematically. 


Providers’ Backgrounds and Experience 

The four providers selected for WorkAdvance had a range of backgrounds. Ultimately, this 
benefited the study, as it allowed the study to leam, among other things, whether a variety of 
providers — including some that were less experienced in their delivery of sectoral programs 
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than those included in the earlier Sectoral Employment Impact Study (SEIS) 3 or that had no sec- 
toral experience at all — could successfully implement WorkAdvance. As described in Chapter 
1, the following four organizations were selected to operate WorkAdvance: 

• Per Scholas in the Bronx, New York, which focused on the information 
technology sector 

• St. Nicks Alliance in Brooklyn, New York, which focused on environmental 
remediation and related occupations 

• Madison Strategies Group in Tulsa, Oklahoma, which focused on the trans- 
portation and (later) manufacturing sectors 

• Towards Employment in northeast Ohio, which focused on the health care 
and manufacturing sectors 

This section provides more infonnation about the providers, with an emphasis on their experi- 
ence running sector programs within the location and sector that they operated for WorkAd- 
vance. 4 


Per Scholas 

Per Scholas is a nonprofit infonnation technology training and employment services 
provider. Launching and operating the WorkAdvance program components took the least effort 
for Per Scholas, which already had substantial experience with sectoral programs and evalua- 
tions, including in the SEIS. Per Scholas had essentially already operated all of the main com- 
ponents of WorkAdvance since 1998, with the exception of career advancement and post- 
employment services. The content of technical assistance there primarily concerned adding 
these two domains to what Per Scholas was already doing. 5 

When it came to establishing strong employer relationships, Per Scholas had a head 
start, as it had already been operating a sector program in infonnation technology and had im- 
portant employer relationships in place, though it also developed new ones for WorkAdvance. 
However, Per Scholas ’s emphasis had primarily been on getting its graduates into entry-level 
infonnation technology jobs, not assisting with career advancement. Regardless of this shift in 
emphasis, Per Scholas maintained a strong business- facing focus; never was there a reference to 
Per Scholas graduates being “program participants” or “low income.” 


Maguire et al. (2010); see Chapter 1 of the present report. 

4 For more extensive background on the providers, as well as more information on the early implementa- 
tion of WorkAdvance, see Tessler et al. (2014). 

5 For all providers, technical assistance was provided by MDRC, and sometimes by contractors working 
closely with MDRC. 
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St. Nicks Alliance 

St. Nicks Alliance is a large multiservice agency housed in several locations in Wil- 
liamsburg, Brooklyn, offering help with affordable housing and health care, workforce devel- 
opment services, and youth services, among other social programs; WorkAdvance was a small 
part of the organization and was largely staffed by individuals already working within St. Nicks 
Alliance. This provider had to make the biggest shifts in order to implement the WorkAdvance 
model. Although St. Nicks Alliance had operated specific vocational training programs for 
more than 10 years, including one in environmental remediation, it had to develop a sectoral 
strategy for its skills training program; diversify its sectoral effort to include related occupations, 
such as pest control; and add a focus on advancement and on working with employers in more 
of a demand-driven way. These changes took time. Additionally, the staff members managing 
WorkAdvance were balancing its needs with the need to attend to St. Nicks Alliance’s other 
workforce development programs. 

St. Nicks Alliance had developed some fruitful relationships with employers within the 
environmental remediation sector before WorkAdvance, but it was more accustomed to provid- 
ing case management to address participants’ employment barriers and less accustomed to treat- 
ing employers as a second customer. Staff members working on WorkAdvance initially were 
not well connected to employers’ needs and relied on a small network of employers for place- 
ments, rather than attempting to expand that network. Eventually, with technical assistance, the 
number of these connections started to grow somewhat, but St. Nicks Alliance’s overall orienta- 
tion toward workers rather than employers did not change much over time. 

Madison Strategies Group 

Madison Strategies Group is a nonprofit spinoff of Grant Associates, a for-profit work- 
force development company that operates a variety of programs in New York City, including 
sector strategies. Madison Strategies Group transferred its parent organization’s considerable 
sectoral experience from New York City to Tulsa, where it had never operated. Madison Strate- 
gies Group initially offered services only in the transportation sector, including training for 
transportation-related manufacturing; however, the manufacturing focus gradually became dis- 
tinct from transportation as it became clear that someone who is trained to manufacture trans- 
portation-related parts has the requisite skills to work in manufacturing more generally. It is 
now most accurate to say that Madison Strategies Group focuses on both the transportation and 
manufacturing sectors. 

While its leaders’ sector experience helped Madison Strategies Group understand what 
it would take to launch and operate WorkAdvance, it was at a disadvantage relative to the other 
providers in that it was developing completely new operations in a new location, including hir- 
ing a staff, making connections with training providers, and developing brand new employer 
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relationships. Additionally, while its leaders had a deep understanding of the transportation sec- 
tor from their work in New York City, when Madison Strategies Group expanded into manufac- 
turing it had to develop knowledge of that sector and forge new relationships with employers — 
in addition to learning the Tulsa-specific nuances of the transportation sector. That said, because 
Madison Strategies Group was singularly focused on WorkAdvance and had sector experience 
through its relationship with Grant Associates, it was able to establish WorkAdvance services 
more efficiently than would have been likely if it had also been operating other programs or was 
new to sector programs in general. 

Towards Employment 

Towards Employment is a community-based organization that provides a broad range 
of employment services for low-income populations in Greater Cleveland. Towards Employ- 
ment focused on the health care and manufacturing sectors in WorkAdvance, and it had its own 
challenges in implementing the program. While it was an established organization in its com- 
munity and had extensive experience with workforce and work readiness programs, particularly 
in the health care sector, it had not operated sector programs before, and its experience was lim- 
ited primarily to entry-level nursing assistant and other low-skill long-term care positions. In 
addition, many existing staff members had backgrounds primarily in general employment issues 
and the provision of social services. Finally, it had to leam about the manufacturing sector and 
had to launch and develop WorkAdvance services in two sectors and in two locations . 6 

In an arrangement probably typical of many sector programs, Towards Employment 
managed multiagency collaborative partnerships in both locations with organizations that were 
responsible for delivering different aspects of the WorkAdvance services, allowing Towards 
Employment to draw on, and not duplicate, existing service expertise in the area. In addition, 
Towards Employment affiliated with industry intermediary groups that brought their own in- 
dustry knowledge and relationships with targeted employers to the delivery of WorkAdvance 
services. The organization managed these complex relationships while operating other pro- 
grams at the same time. Even under these circumstances, Towards Employment put all the 
WorkAdvance pieces in place. 


WorkAdvance’s Services 

This section describes the recruitment, screening, and enrollment of the WorkAdvance research 
sample and the implementation of each component of the WorkAdvance model. It also de- 
scribes the extent to which enrollees actually engaged in the services offered. Broad descrip- 


6 As mentioned in Chapter 1, the Compass site in Ohio was not included in the evaluation’s analysis. (See 
Tessler et al., 2014.) 
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tions are provided; site-specific practices are mentioned only when a site’s practice differed sig- 
nificantly from those of the other sites, or as examples. 

Recruitment and Screening 

• WorkAdvance had fairly intensive screening of program applicants. In- 
dividuals who eventually enrolled in the program, however, still faced 
substantial barriers to employment. 

The WorkAdvance recruitment and screening process sought to identify candidates 
who had the ability to complete training successfully and be attractive to employers, while not 
being so qualified that they could find appropriate employment on their own and thus would not 
need the program’s services. In addition, unlike many other workforce programs that may have 
only basic eligibility requirements such as income guidelines, WorkAdvance required that can- 
didates demonstrate an interest in and commitment to the particular sector, have the ability (as 
perceived by WorkAdvance staff members) to work in that sector, and have the motivation 
(again, in the view of the program staff) to complete the required training. 

As a result, the WorkAdvance providers screened applicants fairly intensively. Program 
administrators used a combination of objective criteria (such as income guidelines and reading 
and math literacy test scores) and subjective criteria (such as staff views of applicants’ attitudes, 
interest in training and likelihood of completing it, and desire for advancement) to screen 
WorkAdvance applicants for program suitability. Nevertheless, many enrolled individuals faced 
substantial barriers to employment; for example, long-tenn unemployment, previous criminal 
convictions, unresolved legal matters, and unstable housing. 

• About one in five program applicants enrolled in the study and were 
subsequently randomly assigned. The vast majority of applicants who 
were screened out did not meet objective academic criteria or failed to 
attend enrollment-required meetings. 

Among individuals who initially expressed interest in WorkAdvance, a minority — 
from about 35 percent at St. Nicks Alliance to only about 8 percent at Madison Strategies Group 
— were both still interested in and qualified for enrollment by the end of the intake process. 
(See Figure 2.1.) It is important to note, however, that relatively few applicants were screened 
out as inappropriate by more subjective criteria at the discretion of the providers’ staff. Most of 
the individuals who did not eventually enroll in the program either withdrew on their own ac- 
cord or failed to achieve the required score on assessments of their academic level. The thor- 
ough intake process, which could span several days, turned out to be a key feature of program 
implementation, despite occurring before random assigmnent. For example, the up-front 
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The WorkAdvance Study 
Figure 2.1 

Percentage of Applicants Remaining, by Step in Intake Process 



SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas and St. Nicks 
Alliance, a report based on program tracking data provided by Madison Strategies Group, and program tracking 
data provided by Towards Employment. 

NOTE: Refer to Appendix Tables C.l and C.2 for provider-specific details regarding the length and timing of the 
data coverage period and the definition of an applicant, as there is variation across providers. 


screening for applicants’ motivation is probably one of the reasons for the high rates of partici- 
pation in program activities, discussed below. 

Since it was understood from the outset that some individuals would be screened out 
before random assignment, the providers sought to make recruitment efforts as efficient as pos- 
sible. Efficiency was especially important because the providers were filling class cohorts to 
start training together; the longer it took to recruit eligible and qualified individuals, the longer 
those who were enrolled and ready to start training had to wait for their classes to begin. 
Sources that brought in the largest number of applicants for WorkAdvance — for example, 
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friends and family members and the Internet — were not always the same sources that brought 
in the largest pool of eligible and qualified applicants, depending on the provider, and an aware- 
ness of this distinction helped providers direct their outreach efforts more productively. (See 
Appendix Table C.l.) 

• Recruitment and screening procedures did not differ substantially 
across the providers. 

Each provider customized recruitment and screening nuances to its specific targeted 
sectors, but no provider stood out from the others in level of intensity of scrutiny. (See Appen- 
dix Table C.2 for the intake procedures by provider.) All providers refined their screening 
processes over time, based on employer feedback. For instance, Per Scholas eliminated some 
questions from the Test of Adult Basic Education that assessed skills that were not of high in- 
terest to information technology employers. In another example, St. Nicks Alliance added a 
criminal background check to the environmental remediation technician training program intake 
process, because potential employers were eventually screening out candidates with such a 
background. 

Informally, WorkAdvance program managers across all four providers mentioned that 
toward the “end” of the study enrollment period, which each defined differently (anywhere 
from the last six months to the last month or so of enrollment), they relaxed their screening 
standards somewhat in order to get the required number of people into the study by the sample 
enrollment deadline. 7 While conversations with program managers suggested that they allowed 
less “qualified” individuals into the program than they would have preferred — for example, 
people who may not have showed the desired commitment or motivation or may not have pre- 
sented themselves in the best light — a comparison of demographic data for enrollees who en- 
tered WorkAdvance in the last few months of the study enrollment period with data for enrol- 
lees who entered WorkAdvance earlier did not reveal stark differences, and differences that did 
emerge did not go in a consistent direction: Some comparisons suggested that late enrollees 
might have been more qualified than earlier ones; others suggested the opposite. The program 
managers’ impressions, however, could still be true, as the demographic data do not shed light 
on enrollees’ relative levels of commitment, motivation, or other more subjective factors. 

Once a potential program participant passed through the screening process, he or she 
was randomly assigned to either the WorkAdvance group or the control group. WorkAdvance 
group members’ subsequent flow through the WorkAdvance program varied by provider. A 
principal reason for this variance is that, as discussed in Chapter 1, Towards Employment and 

7 As a result of the difficulties that the providers encountered with recruitment, study sample goals were 
revised downward several times. The total sample goal across providers was originally 3,850, to be enrolled 
over a two-year period (mid-2011 to mid-2013); in the end, it was reduced to 2,600, or between 500 and 700 
individuals per provider. 
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Madison Strategies Group initially had two different program tracks: one that offered training 
first and another that placed people directly into jobs first, with fonnal or informal training to 
follow. In contrast, all WorkAdvance group members at Per Scholas and St. Nicks Alliance 
went into training first. Individuals assigned to the control group, regardless of provider, were 
given infomiation about the local workforce system, including the American Job Centers net- 
work (One-Stop Career Centers), where they could seek information about training opportuni- 
ties. 


Preemployment and Career Readiness Services 

In programs such as WorkAdvance, many participants need more than just vocational 
training to find a job and succeed in it. The WorkAdvance model therefore included a career 
readiness component, which included classes to teach WorkAdvance participants about the des- 
ignated sectors and help them acquire the resume and the interview skills needed to be hired and 
the “soft skills” critical to success in the sectors, as well as preemployment coaching to help 
participants set and follow through with career advancement goals (Table 2.2). 8 

• Almost all WorkAdvance group members engaged in career readiness 
activities, and the basic content of such activities was similar across pro- 
viders. 

As Table 2.3 indicates, WorkAdvance group members’ engagement in career readiness 
activities was impressively high: Combining all the providers, close to 95 percent of WorkAd- 
vance group members engaged in a career readiness activity. 

All providers covered such topics as an introduction to the sector; how to get a job in 
the sector, including how to develop a resume and cover letter; job search ideas and how to pre- 
pare for job interviews (one provider, for example, had all WorkAdvance participants prepare 
and perfect over time a one-minute pitch to employers); and the development of individualized 
career plans (ICPs) 9 — although some emphasized certain topics more than others. 10 Some also 


8 "Soft skills” refers to the general habits and competencies that make for an effective employee, such as 
showing up to work on time and working cooperatively with others. 

9 While seemingly a good tool for first-time planning, the ICP in practice was not the "living document” 
that it was intended to be; it was mostly not updated after it was initially completed. However, staff members 
did use the ICPs sometimes to reengage participants and restart conversations with those whose contact with 
their coach had lapsed. 

10 Despite attempts to customize some of the curricula to the targeted sector, feedback from participant in- 
terviews suggests that career readiness classes largely covered general topics. Preparation for work in a given 
sector appeared to be covered more thoroughly and more intensively in orientations that the providers offered 
— sometimes as stand-alone activities and sometimes as the first session of career readiness classes — than in 
the classes themselves. 
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The WorkAdvance Study 


Table 2.2 

Career Readiness Services, by Provider 



Per Schoias 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Career readiness services 

Sector 

orientation 

Stand-alone 1 -day kickoff; 
also integrated into CRT 

Integrated into CRT 

Integrated into CRT 

Integrated into CRT 

Career 

readiness 

training 

Concurrent with OST 

Concurrent with OST 

Typically provided for 1 week 
immediately following 
enrollment and before start of 
OST 

Placement-first: typically 
provided for 2 weeks before 
start of job search; training- 
first: typically provided 
concurrent with OST 

Hours of CRT 

12 sessions at 7 hours each 

9 sessions at 4 hours each 

5 sessions at 6 hours each 

10 sessions at 6 hours each 

Transportation 

assistance 

Need-based 

Need-based 

Bus passes or gas cards are 
provided to all training 
participants; otherwise need- 
based 

Need-based 

Preemployment 

coaching 

One-on-one sessions during 
OST. Coaches follow up with 
e-mails or in person during job 
search. 

One-on-one sessions during 
OST. Coaches are expected to 
follow up 2-3 times per week 
during job search. 

One-on-one sessions during 
CRT. Coaches visit weekly 
with participants as a group 
during OST. One-on-one 
coaching during OST as 
needed. 

Coaches introduce career plan 
during CRT. Coaches meet 
with participants in groups 
and one on one up to 3 times 
per week during job search. 


SOURCES: Documentation supplied by providers and interviews with provider staff. 
NOTE: CRT = career readiness training; OST = occupational skills training. 




The WorkAdvance Study 
Table 2.3 


Participation in Career Readiness Activities and Supportive Services 
Within 18 Months of Random Assignment, Cross-Site 


Career readiness activity since RA (%) 

Per 

Scholas 

St. Nicks 

Alliance 

Madison Towards 

Strategies Group Employment Overall 

Ever participated in any career readiness activity 8 

95.7 

82.6 

98.3 

98.3 

94.7 

Started classroom-based career readiness training 13 

95.7 

82.6 

98.3 

70.2 

87.1 

Completed classroom-based career readiness 
training 

78.5 

78.9 

92.1 

62.8 

78.0 

Received help obtaining supportive services 0 

49.0 

NA 

72.2 

71.4 

64.2 

Sample size 

349 

242 

353 

349 

1,293 


SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 
Strategies Group (MSG), and Towards Employment (TE). 

NOTES: RA = random assignment; NA = not applicable. 

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. 

Only career readiness activities before first placement are shown in the table. These might include attending 
post-RA orientation or assessment, starting career readiness training, developing an individualized career plan, 
receiving help with resume development and job applications, and interview scheduling or preparation. 

a The first program activity at PS and SNA is skills training, which is offered in combination with career 
readiness training and includes help with developing a career plan, resumes, and interview preparation. At MSG, 
the first program activity is career readiness training. At TE, the first program activity is an initial assessment 
wherein career goals and barriers to employment are discussed. 

b Some participants may engage in career readiness training with the provider outside a classroom setting. At 
TE, these participants are counted as ever having participated in any career readiness activity. 

c lnformation on supportive service take-up was not included in program tracking data provided by SNA. The 
overall rate for this measure is among participants from PS, MSG, and TE. 


focused more than others on workplace skills such as punctuality, teamwork, and problem- 
solving. All the providers had to adjust to the WorkAdvance model idea that messages about 
planning for advancement had to be infused from the beginning of enrollees’ engagement with 
WorkAdvance and not treated as an add-on after enrollees became employed. It took a while 
before providers addressed advancement from the start of service delivery — for example, by 
having the occupational skills instructors talk about the different types of skills that would be 
required as someone advanced within a particular industry, or by having instructors work with 
participants to think about career ladders rather than just a first job and incorporate these ad- 
vancement goals into their ICPs. 
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Preemployment coaching — one aspect of the career readiness activities — was fo- 
cused on a range of issues, some more technical in nature and some more emotional or behav- 
ioral. Activities and discussion topics included reviewing resumes and preparing for job inter- 
views; discussing anxiety about going back to work; helping participants weigh pay from 
employment against unemployment benefits; addressing unrealistic expectations of wage rates; 
reminding participants to take into account commuting time and costs when considering job 
offers; encouraging discouraged job seekers; preparing individuals to retake certification exams; 
and identifying how to sell “transferable” skills to potential employers. 

• Providers differed in the schedules and length of their career readiness 
training, the curriculum used, the instructors’ experience, and the 
amount of advancement coaching provided, as well as in the degree to 
which workplace norms were modeled by the WorkAdvance staff. 

The provider that approached this program component most distinctively was Per Scho- 
las. Relative to the other providers, Per Scholas placed more emphasis on staff teaching and 
modeling workplace norms and holding WorkAdvance participants accountable to an expected 
set of behaviors. 11 It also had a longer series of career readiness training classes and more career 
coach contact with WorkAdvance participants; its career readiness classes were concurrent with 
occupational skills training and conducted in the same location as other WorkAdvance activi- 
ties, allowing more opportunities for staff members to interact with participants. 

• Employer partners influenced and were engaged in career readiness ac- 
tivities in a variety of ways: via counseling to providers on curricula and 
sector trends, as guest speakers in classes, as mock interviewers of par- 
ticipants, and as hosts for participant workplace visits. 

Madison Strategies Group and Per Scholas, for example, used advisory groups to coun- 
sel them on curricula for career readiness classes as well as on occupational skills training and 
the latest trends in the sector. Madison Strategies Group also hired transportation industry con- 
sultants who came in every week to speak with WorkAdvance participants about the industry. 
Towards Employment had relationships with individual employers and developed relationships 
with existing business intermediary groups in the health care and manufacturing sectors, which 
carried out similar functions. St. Nicks Alliance relied primarily on relationships with individual 
employers in the environmental remediation sector to get input about its career readiness activi- 
ties. The providers had a range of employer partners who came to the organizations’ offices to 
talk about their companies or conduct mock interviews, and employers also hosted visits that 
gave program participants firsthand exposure to the work environment that they could expect to 


n For example. Per Scholas enforces a “zero tolerance” policy for the first three weeks, in which a partici- 
pant can be dismissed from the program for arriving late to class. 
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join. As expected, all the providers adjusted their career readiness services over time, based at 
least in part on input from employers. Changes included Per Scholas’s addition of a customer 
service component, Madison Strategies Group’s use of actual employer interview questions to 
reinforce best practices in applying for a job, and Towards Employment’s integrated examples 
of real job descriptions and performance reviews. 

Occupational Skills Training 

Occupational skills training in WorkAdvance was intended to meet the needs of both 
program enrollees — by helping them to obtain industry-recognized credentials and develop 
technical skills relevant to the targeted sectors and occupations within those sectors — and par- 
ticular local employers. The providers selected their training offerings based on local industry 
demand, staff and institutional knowledge of the industry, and the characteristics of targeted 
occupations — including job entry requirements, pay rates, benefits, and opportunities for ad- 
vancement. Table 2.4 summarizes the key features of the training courses offered by the Work- 
Advance providers. 

• The length, location (at the WorkAdvance site or off-site), and breadth 
of the occupational skills training options differed by provider. 

The length of training, by design, ranged from 2 weeks to 32 weeks, depending on the 
occupation, and was, for the most part, “hands-on” in nature. All four WorkAdvance providers 
generally offered training in cohorts — that is, groups of participants went through training cy- 
cles together. All the providers offered full-time training during regular business hours, though 
Towards Employment and Madison Strategies Group eventually also offered evening and part- 
time classes (both in cohorts and in “mixed” classes), in part to accommodate participants who 
worked during the day. 

In addition to a range of training course durations, providers varied in terms of where 
training was held (on-site or through a contract with an off-site training provider) and the 
breadth of the offerings. Per Scholas was unique in offering all of its training on-site and with its 
own instructors. St. Nicks Alliance had a blend of off-site and on-site trainings (with inde- 
pendently contracted instructors), and Towards Employment and Madison Strategies Group 
partnered with off-site training institutes, including community and technical colleges, with 
some of the courses specifically tailored at the request of these providers (Table 2.4). While the 
evaluation did not explore indicators of the quality of training or whether training quality dif- 
fered in terms of the location, training provided on-site (at the location where other WorkAd- 
vance services were provided) may have had some “spillover” advantages: allowing for more 
contact between the WorkAdvance program staff (including career advancement coaches) and 
the participants, and making in-house instructors generally more aware of the overall mission of 
WorkAdvance, which could have reinforced messages about the importance of planning for 
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Table 2.4 


Occupational Skills Training, by Provider 



Per Schoias 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Occupational skills training 

Duration 

15 weeks 

5 to 12 weeks 

4 to 32 weeks 

2 to 17 weeks 

Training 

courses 

offered 

A Plus, Network Plus 

Environmental Remediation 
Training, Commercial Driver's 
License B (CDL-B) with 
hazmat endorsement. Pest 
Control Technician 

Aviation Manufacturing, 
Commercial Driver's License 

A and B (CDL-A, CDL-B), 
Computerized Numerical 

Control (CNC) Machining, 
Diesel Mechanic, Welding, 
Supervisory Leadership 

MF: Computerized Numerical 
Control (CNC) Machining, 
Welding 

HC: Phlebotomy, Certified 
Health Care Access Associate, 
Patient Care Assistant, State- 
Tested Nurse Assistant, 

Medical Billing and Coding 

Target 

occupations 

Help Desk Technician, IT 

Field Technician 

Environmental Remediation 
Technician, Tanker or Hazmat 
CDL Driver, Pest Control 
Technician 

Aviation Manufacturing 
Assistants, Semi- and Truck 
Driver, CNC Operator, Diesel 
Maintenance Technician, 

Welder 

MF: CNC Operator, Welder 

HC: Phlebotomist, Patient 
Access Specialist, Patient Care 
Assistant and State-Tested 

Nurse Assistant, Certified 
Professional Coder 

On-site/off- 

site 

On-site 

On-site for pest control 
training; mix of on- and off- 
site at private schools for all 
other training courses 

Off-site at private or technical 
schools or community colleges 

Off-site at private or technical 
schools or community colleges 


SOURCES: Documentation supplied by providers and interviews with provider staff. 
NOTES: MF = manufacturing; HC = health care, IT = information technology. 




career advancement that participants were hearing from their career readiness instructors. It was 
also probably easier to customize the training according to feedback from employers when the 
instructors were directly employed by the program; with contracted training, customization re- 
quired more negotiation. 

The providers also used a number of approaches, and invested substantial resources, to 
ensure training completion. Their efforts, which varied by provider, included offering paid in- 
ternships, covering transportation costs, adjusting training schedules to allow for evening clas- 
ses so that participants could work during the day, and providing activities to get families in- 
volved and foster support among them. (Specifically, Madison Strategies Group convened open 
houses at training locations so families could see the equipment their family members would be 
using, understand what they would be doing, and feel more involved and supportive.) Providers 
also checked in regularly with participants and instructors to track progress, including visiting 
off-site training locations on a regular basis. Reasons participants dropped out of training, 
gleaned from case notes, included, for example, an immediate need for income because unem- 
ployment insurance ran out sooner than expected; work schedule changes that no longer al- 
lowed the participant to attend training; and behavioral or mental health setbacks, such as debili- 
tating episodes of depression. The providers tried to work around these issues by rescheduling 
participants for later cohorts or helping them to find employment in the sector. 

• Rates of participation in and completion of training were high across the 
providers, and over half of all enrollees received a nationally or locally 
recognized training credential — a critical first step toward getting a job 
in a sector. 

Within 18 months of random assignment, 87 percent of all WorkAdvance group mem- 
bers had been scheduled for occupational skills training; about 76 percent of all WorkAdvance 
group members actually started the training; and, among those who started training, few — only 
16 percent — dropped out (Table 2.5). These high training participation and completion rates 
probably reflect a number of factors: the extensive upfront screening done in WorkAdvance, 12 
in part to gauge motivation but also to enroll individuals who had the academic skills to handle 
the training; enrollees’ relatively high levels of pretraining education; the peer support offered 
through WorkAdvance’s cohort-based model; appealing instructors and engaging material; and 
the level of effort that staff members put into supporting participants while they were in skills 
training. 


12 Full-time training courses meant that participants had to find ways to support themselves financially. At 
Per Scholas, the screening process required applicants to have financial support that would enable them to 
complete a 15-week, full-time training cycle. As noted above, Towards Employment and Madison Strategies 
Group offered some evening and part-time classes, which allowed participants to work while also participating 
in training. 
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Table 2.5 


Participation in Skills Training Activities Within 
18 Months of Random Assignment, Cross-Site 


Skills training activity since RA 

Per 

Scholas 

St. Nicks Madison 

Alliance Strategies Group 

Towards 

Employment Overall 

Ever scheduled to start skills training (%) 

100.0 

100.0 

78.2 

73.4 

86.9 

Ever started skills training 

95.7 

82.6 

62.0 

65.3 

75.9 

Ever completed skills training 

78.5 

78.9 

48.7 

50.1 

62.8 

Among participants who started skills training 

82.0 

95.5 

78.5 

76.8 

82.8 

Ever dropped out of skills training 

16.6 

3.7 

12.5 

12.9 

12.1 

Among participants who started skills training 

17.4 

4.5 

20.1 

19.7 

15.9 

Enrolled in skills training at 18 months after RA 

0.6 

0.0 

0.9 

2.3 

1.0 

Among participants who started skills training 

0.6 

0.0 

1.4 

3.5 

1.3 

Ever obtained a credential in targeted sector 1 

72.5 

73.6 

48.7 

31.8 b 

55.2 

Average number of weeks in skills training 0 (#) 

13.8 

6.8 

9.4 

9.1 

10.0 

Among participants who started skills training 

14.4 

8.2 

15.1 

13.9 

13.2 

Sample size 

349 

242 

353 

349 

1,293 


SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison Strategies 
Group (MSG), and Towards Employment (TE). 

NOTES: RA = random assignment. 

Italics indicate the metric is not among the foil sample shown in the table. 

Skills training activity refers to enrollment in an occupational skills training course in the targeted sector or 
industry. Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. 

The results in this table are derived from providers' management information system data. Other measures of 
training participation and completion, shown elsewhere in this report, are based on survey data and present different 
estimates. It is common for self-reports from survey data to differ from program records for numerous reasons, 
including problems recalling the timing of events or activities. 

“Credentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 
reporting of locally recognized credentials obtained in the targeted sector. 

b TE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 
obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 
having obtained a credential. 

c The scheduled lengths of training varied across sites, industries, and types of training. 


According to the providers’ MIS data, those who started training generally began it 
within a few months of random assignment and generally completed it within a year of random 
assignment. Across all providers, individuals who started skills training participated in it for an 
average of over 13 weeks (commonly full-time) during the follow-up period (Table 2.5). 
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Provider-specific rates of starting and completing skills training are not clearly correlat- 
ed with any of the differences in training setup described above; that is, such factors as length or 
location of training do not seem to be correlated with rates of starting or completing training. 
Rather, differences across providers in these rates are primarily driven by the two-track ap- 
proach used by two providers (Madison Strategies Group and Towards Employment) before 
summer 2012, which allowed more WorkAdvance participants to be placed directly into jobs 
rather than engaging in training first. Skills training participation and completion rates at these 
two providers are much higher for the late cohort — after the providers began to direct more 
participants to training before employment — than they were for the early cohort, as discussed 
further below. 13 

In most cases, completion of occupational skills training led to earning either a national- 
ly or a locally recognized credential (or both). (See Table 2.5.) Across all sites, a high propor- 
tion of WorkAdvance group members — 55 percent — earned such licenses or certificates. 
Again, differences across providers in these statistics primarily reflect the two-track approach 
used early on by Madison Strategies Group and Towards Employment, as well as a lack of data 
on locally recognized credentials for one of the providers. 14 

• Overall, the WorkAdvance occupational skills training component was 
informed by and adaptive to local labor markets. 

All providers adapted their skills training curricula at some point, based on labor market 
and employer input, and most altered their course offerings as well. Madison Strategies Group, 
for example, added a week to its computerized numerical control (CNC) machining course be- 
cause employers felt that the participants needed specific additional skills, and it dropped sever- 
al components from the aviation training, based on employer feedback that those skills would 
not be needed. Per Scholas developed a unit that trained on the repair and programming of 
handheld devices, such as tablets and smartphones, and St. Nicks Alliance changed its environ- 
mental remediation certificate offerings in response to high demand for mold remediation skills 
following Hurricane Sandy. Towards Employment added an online learning module to its basic 


13 Statistics on the proportion of program enrollees scheduled for training can illustrate the influence of the 
placement-first track, which was primarily in place for individuals who enrolled in the early cohort, on the like- 
lihood of individuals being directed to training: At Madison Strategies Group, 64 percent of the early cohort 
were scheduled for training within 18 months of their random assignment, compared with 92 percent of the 
later cohort. At Towards Employment, 60 percent of the early cohort were scheduled for training within an 18- 
month follow-up period, compared with 86 percent of the later cohort. 

14 The credentials obtained through the diesel mechanic and CNC machinist training sessions at Madison 
Strategies Group and Towards Employment were unique to the local industry’s employer needs and were cre- 
ated by the WorkAdvance providers, in concert with employers or training providers or both, to abbreviate and 
adapt manufacturing training courses and offer credentials accepted by local employers. Madison Strategies 
Group included the credentials obtained upon completion of this locally recognized course in its rates measur- 
ing the earning of a credential, while Towards Employment did not, though the courses were very similar. 
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machining course, in order to incorporate additional content that was preferred by manufactur- 
ing employers but was not typically offered in shorter machining classes. 

Over time, most of the WorkAdvance providers offered new areas of training and 
dropped others. Madison Strategies Group, for example, began CNC machining training in June 
2012 and welding training in February 2013 — both in response to employers’ and industry 
experts’ input. Likewise, it suspended enrollment for a second training session in aviation and 
aerospace in August 2012 in order to better assess the market need, and subsequently held just 
one more session before discontinuing it. Per Scholas responded to employer feedback by add- 
ing a separate training course for program graduates that focused on software testing. St. Nicks 
Alliance added pest control technician training in August 2012, when the demand for environ- 
mental remediation technicians was decreasing and demand for pest control technicians was 
confirmed. Finally, Towards Employment adjusted the types of welding credentials that were 
targeted, based on employer needs, and stopped recruiting for health care patient navigators, 
since the occupation was not growing at the pace initially projected. 

Job Development and Placement Services 

• WorkAdvance job developers served as a crucial link between the pro- 
gram and employers, to maximize the chances that program partici- 
pants would find jobs that offered good pay, security, and advancement 
opportunities. 

Identifying jobs that offer good wages, employment security, and opportunities for ad- 
vancement requires an understanding of local labor markets and the specific needs of employ- 
ers. 15 Moreover, prior workforce studies suggest that sector-based training programs that have 
direct relationships with employers have clear advantages. 16 Job developers thus serve an im- 
portant role as conduits through which program managers receive timely feedback from em- 
ployers and local labor market information so that they can prepare program enrollees for the 
best jobs available. 

In WorkAdvance, MDRC provided substantial technical assistance in some cases to 
help job developers establish closer relationships with employers and industry associations. 17 At 
Madison Strategies Group and Towards Employment, the technical assistance was “lighter 
touch” and focused on early development of local sector relationships. Conversely, St. Nicks 


15 D’Amico (2006). 

16 Maguire et al. (2010); Henderson, MacAllum, and Karakus (2010); Bloom, Hendra, and Page (2006). 

17 While this did not prompt the technical assistance, several program enrollees, when asked in “open- 
ended” questions in the Year 2 survey about how WorkAdvance could be improved, mentioned a need for 
stronger job placement services, and some expressed frustration that they had completed training and certifica- 
tions without finding a job in the sector. 
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Alliance received extensive, ongoing technical assistance to address low placement numbers 
and strengthen job developers’ skills and placement capacity. There, the goal was to increase 
staff capacity, improve internal reporting mechanisms, provide accountability and general ad- 
vice, and improve the provider’s ability to gather and analyze systematic labor market infor- 
mation. While Per Scholas received technical assistance on other program aspects, it did not 
receive any focused on job development or placement. 

• WorkAdvance providers differed in how they staffed the job develop- 
ment role, with one seeking to hire people with sales experience and an- 
other contracting with people at local industry associations to play this 
role. Additionally, some providers directly matched program partici- 
pants and employers, while others recommended multiple candidates to 
employers for any given job. 

Notably, when initially hiring staff members, Madison Strategies Group put greater 
emphasis than the other providers did on selecting a job developer with sales experience, plus 
experience in hiring, recruiting, and managing others, as this was seen as important to demon- 
strate credibility and encourage confidence among employers. 18 While Per Scholas, St. Nicks 
Alliance, and Madison Strategies Group had full-time on-site staff members devoted to job de- 
velopment and placement, Towards Employment contracted with local industry associations 
and industry-specific workforce intermediaries in Greater Cleveland to establish relationships 
with both the manufacturing and the health care sectors, and career coaches then handled the 
relationships between particular WorkAdvance participants and employers. The goal was to 
make use of these associations’ existing relationships to allow for more rapid acceptance of 
WorkAdvance participants, and the use of a partnership was part of a strategy to encourage col- 
laboration and the spread of WorkAdvance practices across key workforce partners, for the sake 
of sustainability. For the manufacturing sector, this relationship endured, but in early 2012, the 
health care industry association’s priorities changed and that contract was terminated. In fall 
2012, Towards Employment added a different health care industry association as a partner, but 
that relationship also ended. Towards Employment eventually began working directly with 
large hospitals in Cleveland. 

Per Scholas and Madison Strategies Group had strong direct employer relationships and 
conducted individualized job matching between participants and employers. St. Nicks Alliance 
matched individuals to jobs but also identified and recommended multiple candidates for any 
given job. Towards Employment relied on intermediary organizations to match participants 
with job leads. In none of the WorkAdvance sites did providers or participants expect to have a 
job “guaranteed” to those who completed WorkAdvance training. 

Aladison Strategies Group, unlike the other providers, had a performance-based salary structure tied to 
employment outcomes. For job developers with sales experience, this was a familiar way of operating. 
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• WorkAdvance providers dealt with employers with different character- 
istics, in line with the sectors they targeted, which may have influenced 
the likelihood of placing program participants into jobs. 

While the WorkAdvance study was not set up to systematically analyze the depth, 
breadth, or “quality” of the providers’ relationships with employers, there were some notable 
differences in the employers with whom the providers worked. For example, most employers of 
information technology workers are large, “corporate,” and “professional” in nature; they in- 
clude technology and telecommunications finns but also banks, universities, hospital systems, 
and other large corporations that have an information technology unit. This may have made job 
placement efforts somewhat more efficient at Per Scholas, in that many people could be placed 
at the same company (although Per Scholas did not work exclusively with large employers). It 
is also important to note that Per Scholas has a long history in the sector and is known by many 
employers (in contrast to some of the other providers who had to develop these relationships). 19 
Strong relationships between Per Scholas and a number of these large finns may have opened 
doors to opportunities that participants might not have been able to get on their own — assisted 
by the fact that Per Scholas offered a package of training and career readiness services in infor- 
mation technology that was not easily found elsewhere. In the health care sector, the primary 
employers were hospitals; while these were also large, “professional” employers, it is possible 
that Towards Employment was competing with many other training programs focused on health 
care jobs. In addition, hospital jobs are generally more accessible to low-income individuals 
seeking employment in lower-level jobs, compared, for example, with jobs at large financial 
corporations. 

Manufacturing and environmental remediation employers (such as those served by St. 
Nicks Alliance and Towards Employment) tended to be smaller to midsize shops, some of 
which were family-run businesses, which had fewer available job openings on an ongoing basis. 
Thus, these sectors required more intensive job development efforts. Given that the smaller em- 
ployers did not have as many positions, and thus job openings were not as frequent, opportuni- 
ties probably were not as common for ongoing or long-lasting relationships to develop between 
the WorkAdvance providers and the employers, compared with the situation with large employ- 
ers. That said, an advantage for providers working with small- to medium-sized employers was 
that these employers often lacked large human resource departments, and thus the employers 
may have more readily perceived a need for WorkAdvance’s services. It may also have been 


19 This chapter does not present statistics on employment, as data from the Year 2 Survey provide a much 
more complete and accurate picture of employment than statistics based on the reporting of jobs to WorkAd- 
vance staff. Using Year 2 Survey data. Chapter 5 details the types of jobs WorkAdvance group members 
found, as well as the extent to which they were employed over an 18-month follow-up period. The analysis in 
Chapter 5 also indicates the extent to which the jobs found by WorkAdvance enrollees were in the sector in 
which they had trained and offered potential for advancement. 
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easier for WorkAdvance providers to adjust their emphases in response to labor market shifts 
when working with more, and smaller, employers — the closing of a few smaller employers 
would not have required as big a shift for the WorkAdvance provider as the closing of very 
large employers. 

Employers’ reactions to the WorkAdvance model were hard to gauge. See Box 2.1, 
however, for the views expressed in a small number of employer interviews regarding employ- 
ers’ perceived value of the program. 

Postemployment Retention and Advancement Services 

Simply helping workers find employment is often not enough to improve earnings in 
the long run; in many cases, the kinds of jobs people find (generally low- wage ones) do not of- 
fer opportunities for hard-skill acquisition, and these skills are often needed in order for earnings 
to increase. 20 In light of this, programs that have been studied have experimented with different 
ways to provide services to workers after they obtain employment to help them advance over 
time. 21 To help workers figure out the best way to advance — for example, whether and how to 
try to move up at a current employer, apply for a higher-level position at a different employer, 
or obtain additional credentials first — it stands to reason that coaches should provide concrete 
guidance, including setting specific goals, sharing feedback from employers on performance, 
and getting permission from participants to hold them accountable for their commitments. 
Drawing on suggestions from previous studies about what appears to work and what does not to 
help low-wage workers advance, the postemployment services in WorkAdvance were designed 
to include a follow-up plan to contact and communicate with the employee at strategic points 
after the start of employment; updating of the employee’s individual career plan at least twice 
during the first year of employment, to foster a focus on career advancement rather than only 
job retention; and maintenance of regular contact with the employer. 

• Postemployment services were the last set of services to be completely 
developed and implemented. Once they were fully in place, the Work- 
Advance providers differed in how often the coaching services were of- 
fered, in whether incentives were offered to staff members for connect- 
ing with program participants, and in their advice regarding how 
quickly or the best ways in which to seek advancement opportunities 
(which differed by sector). 


20 Hendra et al. (2011); Gladden and Taber (2009); Card, Michalopoulos, and Robins (2001); Miller, 
Tessler, and van Dok (2009). 

21 Hamilton and Scrivener (2012b). 
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Box 2.1 

Employer Perspectives on WorkAdvance 

As part of the WorkAdvance evaluation, a limited number of interviews were conducted with em- 
ployers who worked with WorkAdvance providers and had hired WorkAdvance participants. 
These interviews suggested, as illustrated below, that employers particularly valued several as- 
pects of the program: its identification of individuals who showed a commitment to a career in a 
particular field, its emphasis (at one provider) on modeling workplace behavioral norms, its abil- 
ity and willingness to adapt skills training to employer needs, and its prescreening of potential job 
applicants. 

Employers appreciated applicants’ interest in a career, rather than just a job. As one employer in 
northeast Ohio stated, “The message that resonated with me was the fact that Towards Employ- 
ment had the intention of getting people training who are interested in growing in the manufactur- 
ing industry and who are planning on making a career in manufacturing. . . . We are always look- 
ing for people who want to grow within the industry and are not interested in those who just want 
to get any job.” 

Modeling workplace norms and holding participants accountable to an expected set of behaviors 
was emphasized at one WorkAdvance provider, Per Scholas, and this practice seemed to make a 
good impression on employers. As one employer said, “The school’s always clean, everyone’s 
professional and nice. They get on their students. If anyone is late to class, or not dressing appro- 
priately, they get on them. It’s that that really takes individuals from the community and takes 
them to the next level.” 

All WorkAdvance providers adapted their skills training curricula at some point based on labor 
market and employer input. Both Towards Employment and Madison Strategies Group, for ex- 
ample, worked with employers or training providers to abbreviate or expand manufacturing 
courses. Madison Strategies Group added a week to a machining class because employers felt the 
participants needed specific additional skills, and dropped several components from its aviation 
training based on employer feedback that those skills would not be needed. One employer who 
works with Madison Strategies Group noted in an interview that he appreciated that he “could 
help frame the curriculum to be in line with what we are looking for” and that the provider could 
“tailor make the product we need to fill the voids” in positions that they have had difficulty fill- 
ing. Similarly, one employer that works with Per Scholas noted that “Per Scholas is always eager 
to get feedback to help develop the students’ skill sets and interviewing. ... Positive or negative, 
they always want to hear it.” 

Interviewed employers also noted WorkAdvance ’s value in prescreening job applicants and referring 
individuals who already had the required certifications. One employer said he needs to spend a week 
framing new employees from other recruitment sources, whereas, by working with St. Nicks Alliance, 
“it probably saves us a couple thousand dollars [for] each guy.” Similarly, an employer in Tulsa 

(continued) 
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Box 2.1 (continued) 

reported that Madison Strategies Group’s services helped his company “reduce time to hire, re- 
duce turnover, troubleshoot with new hires, reduce safety infractions, and fill skilled positions be- 
yond entry level,” the latter of which had the biggest impact on the company. And one employer 
in northeast Ohio reported that, compared with other recruitment sources he used, Towards Em- 
ployment was more interested in learning exactly what the employer was looking for in a candi- 
date, while other recruitment sources “simply give us anybody” because those other entities “real- 
ly only care about making money for themselves.” 

Several employers did have criticisms. For example, when making hiring decisions, they valued 
“softer” skills, or what one employer called “essential skills” — such as interview skills, good at- 
titudes, and a willingness to work — and they noted that some of the WorkAdvance participants 
they had hired (as well as hires from other sources) had shortcomings in these areas. 


Providers were preoccupied for much of the first year or two of program operations 
with recruiting and enrolling individuals into the study and developing, launching, and adapting 
the other core program components. But to some extent, the delay in fully rolling out the 
postemployment advancement services also reflected, for at least some of the staff, a skepticism 
of the value of the postemployment services and the need to change the culture of their organi- 
zations. The WorkAdvance model required staff members, even after program participants 
found employment, to put a priority on continued service delivery and to value an ongoing rela- 
tionship with the participant — something not typical in workforce programs — and it took 
some time for this perspective to develop among some staff members. Perhaps just as im- 
portant, it also took time and effort, and substantial technical assistance, for staff members to 
gain knowledge of what types of advancement services to provide and how to provide them. 

Once the postemployment component did take hold across the WorkAdvance sites, 
WorkAdvance staff members at two of the providers normally contacted participants during 
their first month on the job and followed up monthly thereafter. Per Scholas was unusual in that 
it followed up less frequently (quarterly) in the postemployment period, although the coaches’ 
preemployment contact with participants was more frequent than that of the other providers. At 
Madison Strategies Group, staff members contacted participants in their first week of employ- 
ment and then monthly thereafter. WorkAdvance staff members also initiated contact with em- 
ployers to see how program graduates were doing on the job. As Table 2.6 indicates, this out- 
reach to employers was frequent at three sites and rare at Per Scholas. 
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The WorkAdvance Study 
Table 2.6 

Postemployment Services, by Provider 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Postemployment services 

Staffing 

1 full-time career coach with 4 
additional coaches/career 
readiness training instructors 
(serving WorkAdvance and 
non-WorkAdvance Per Scholas 
clients) 

1 senior coach with 2 additional 
career coaches/case managers 

2 career coaches 

4 career coaches (2 for each 
sector) 

Initial 

postemployment 

follow-up 

Coaches follow up with 
participants during the first 90 
days of employment. Job 
developers follow up with 
employers in the same period. 

Job developers follow up with 
participants weekly during the 
first 30 days of employment. 

Job developer or career coach 
follows up with participant and 
employer within the first week 
of employment. 

Coaches follow up with 
participants weekly for the 
first 30 days after placement. 

Ongoing follow- 
up 

Coaches follow up with 
participants quarterly, either 
over the phone or in person one 
on one. Participants are also 
invited to workshops on-site. 

Coaches follow up with 
participants monthly, either 
over the phone, in person one 
on one, or in a workshop 
setting. 

Coaches follow up with 
participants monthly, either 
over the phone, in person one 
on one, or in a workshop 
setting. 

Coaches follow up with 
participants bimonthly 
through 90 days, monthly 
until 180 days, and quarterly 
thereafter. 


SOURCES: Documentation supplied by providers and interviews with provider staff. 




Madison Strategies Group was unique among providers for having staff compensation 
tied to performance benchmarks for WorkAdvance staff members in the area of postemploy- 
ment services (and, as discussed above, in the area of employment outcomes). There, staff 
members were expected to try to contact a minimum of 60 percent of active clients and at least 
80 percent of their total caseloads, and compensation varied accordingly. Attempts to reach cli- 
ents were recorded in Madison Strategies Group’s MIS, and the program manager used the MIS 
to monitor perfonnance. The other providers’ program managers also monitored staff efforts to 
contact participants via the providers’ MIS, and set clear expectations for the staff, but there 
were no financial incentives to meet the expectations. 

In the first three months that WorkAdvance participants were on the job, WorkAdvance 
staff members across all providers tended to focus on retention issues while still working with 
program enrollees to set them up for advancement in the future. Providers agreed that an appro- 
priate amount of time had to pass after employment began before enrollees would be open to 
thinking beyond retention toward advancement, and before employers would be willing even to 
discuss moving an employee up. After the first three months on the job, the theme of the inter- 
actions tended to become more focused on advancement-related topics. 

Some aspects of advancement advice given by WorkAdvance staff members varied by 
sector. Generally, providers advised clients to stay on the job for at least six months, though the 
advice they gave regarding the recommended time before looking elsewhere varied by provider 
and sector. In the manufacturing and environmental remediation sectors, staff members reported 
that changing jobs in pursuit of advancement is more common or acceptable. In information 
technology and health care, where job hopping is seen by employers as a more notable blemish 
on a resume, WorkAdvance providers recommended that clients stay on the job for at least a 
year. 


After a considerable amount of technical assistance for some providers (and considera- 
ble effort by program staff members), all the providers had a much greater emphasis on coach- 
ing (defined as working with participants toward their goals by building on their strengths) than 
on case management (defined as focusing on participants’ barriers and the services needed to 
overcome them) — even at providers where staff members were originally trained as case man- 
agers. Three of the providers maintained this distinction by typically referring participants to 
other organizations for services to address barriers; meanwhile, the coaches worked on building 
on participants’ strengths and focused on advancement. 

• A special study suggested that advancement coaching occurred fairly 
frequently, and that most WorkAdvance group members who were en- 
gaged in coaching meetings were employed. Advancement-related topics 
most commonly discussed included opportunities for advancement at a 
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current job or at a new job, and additional education or training oppor- 
tunities. 

To get a clearer sense of how advancement coaching was delivered, MDRC conducted 
a special study that documented the details of advancement advice and suggestions delivered 
during in-person and phone contacts with individual WorkAdvance participants that occurred 
over a specific three-month period. All four WorkAdvance providers documented, from mid- 
April 2014 to mid-July 2014, the mode of advancement coaching, its frequency, and topics dis- 
cussed during all advancement coaching meetings (Tables 2.7, 2.8, and 2.9). 

The advancement coaching study results suggest that advancement coaching occurred 
for around one-third of WorkAdvance participants during this three-month period (Table 2.7), 
and that the majority (or 76 percent) of enrollees who were engaged in a coaching meeting dur- 
ing this three-month period attended at least one coaching session while employed (Table 2.9). 

As Table 2.8 shows, among all individuals who had a coaching meeting during this pe- 
riod, phone meetings were somewhat more prevalent than in-person meetings, and these phone 
discussions were mostly initiated by WorkAdvance staff members, as opposed to WorkAd- 
vance participants. Staff interviews and observations suggested that advancement-related advice 
was conveyed in other ways as well, including via individual emails, e-mail blasts, Linkedln 
groups, and, at Per Scholas, in-person group “new hire” workshop sessions. 22 

The advancement coaching study also suggested that in coaching interactions, topics re- 
lated to job retention and advancement were more commonly discussed (at least during the spe- 
cial study period) among employed than among unemployed meeting attendees. The most 
common retention-related topic discussed during the special study period was verifying em- 
ployment, which was a requirement of the Social Innovation Fund grant that funded WorkAd- 
vance (Table 2.8). 23 Advancement-related topics most commonly discussed included job- 
specific opportunities for advancement and additional education or training opportunities. Case 
note reviews provided more specific examples of topics discussed during advancement coach- 
ing sessions, including (out of a large variety of topics) trying to unearth problems employees 
were possibly having on the job (difficulty handling job responsibilities or tasks, or problems 
with coworkers); encouraging employees to get an employee handbook to learn about employ- 
ers’ expectations and policies — for example, the number of acceptable absences in the first 90 


~ 2 While Per Scholas had fewer one-on-one meetings between graduates and coaches than the other pro- 
viders, it engaged graduates in a variety of other ways following employment: It convened alumni events, 
formed affinity groups on Linkedln and Facebook, held advance-level certification classes for alumni, and sent 
e-mails to graduates about new job opportunities and new certification classes. 

23 Multiple topics were discussed in most coaching meetings, as is implied by Table 2.8. In very few 
coaching interactions was verifying employment/retention the only topic discussed. 
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The WorkAdvance Study 
Table 2.7 


Percentage of WorkAdvance Group Members Who Attended an Advancement 

Coaching Meeting, Cross-Site 


Activity (%) 

Per 

Scholas 

St. Nicks 

Alliance 

Madison 

Strategies 

Group 

Towards 

Employment 

Overall 

Attended an advancement coaching meeting 

10.9 

22.3 

51.0 

42.7 

32.6 

Sample size 

349 

242 

353 

349 

1,293 


SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by 
two sites and in program tracking systems by two sites. 


NOTE: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 
Provider staff members documented all phone and in-person advancement coaching meetings with participants 
during that time period. 


days, the performance review process, and the pay raise schedule; encouraging employees to get 
infonnation about the responsibilities of different jobs; and practicing how to negotiate for 
higher wages. 

• At three of the providers (not Per Scholas), WorkAdvance staff mem- 
bers had direct discussions with employers about particular employees. 

Staff members at three providers engaged “partner” employers — those with whom 
they had a close relationship — in discussions about the advancement of particular participants. 
Examples of this kind of engagement included advocating for higher wages, for example, if a 
participant received an additional certification but the pay was not adjusted accordingly; making 
plans to advance incumbent employees and backfill the positions they would vacate; and inter- 
vening to encourage a discussion between an employee and a supervisor to discuss advance- 
ment. At Per Scholas and Madison Strategies Group, it was the job developers who talked with 
employers about particular participants. Per Scholas managers indicated that the job developers 
had the relationships with the employers, and they felt it would be “bad for business” for the 
coaches to reach out to the employers, because it could inadvertently send the message that Per 
Scholas graduates had issues that the program staff needed to check up on (and that employers 
should worry about). 


55 






The WorkAdvance Study 
Table 2.8 

Mode and Content of Advancement Coaching Meetings for WorkAdvance Group 
Members Who Attended a Meeting, Cross-Site 


Madison 

Per St. Nicks Strategies Towards 

Activity Scholas Alliance Group Employment Overall 


Average number of advancement coaching 
meetings attended 11 
By phone 

Initiated by provider staff member 
Initiated by participant 
In person 

Attended a meeting by phone (%) 

Attended a meeting in person (%) 

Discussed job search-related topic (%) 

Updating individual career plan 
Updating resume 

Identifying skills and career interests 
Job search activities 
Interviewing 

Discussed retention-related topic (%) 

Verifying employment/retention 

Acclimating to job 

Passing probationary period 

Getting an employee handbook/job description 

Managing problems with coworkers/ supervisors 

Backup plans for transportation or child care 

Feedback from employer on job performance 

Discussed advancement-related topic (%) 
Opportunities for advancement at current job 
Asking for a raise, promotion, or more hours 
Acquiring job benefits 
Improving work schedule or shift 
Feedback from employer on advancement 
opportunities 

Opportunities for advancement at new job 
Additional education or training opportunities 
Other advancement topic 

Discussed other topic (%) 

Financial goals 

Work-life balance 

Participant invited to alumni event 

Other topic (none of the abovementioned) 


1.0 

1.7 

1.8 

2.9 

2.1 

0.6 

1.0 

1.0 

2.2 

1.4 

0.5 

0.8 

0.7 

1.4 

1.0 

0.1 

0.2 

0.3 

0.9 

0.5 

0.3 

0.7 

0.8 

0.7 

0.7 

60.5 

74.1 

72.8 

92.0 

78.6 

34.2 

55.6 

50.0 

38.3 

45.1 

36.8 

48.2 

48.3 

64.4 

53.0 

15.8 

11.1 

2.8 

28.2 

14.0 

18.4 

7.4 

20.6 

28.2 

21.4 

5.3 

3.7 

21.1 

35.6 

22.6 

15.8 

37.0 

41.1 

47.7 

40.6 

5.3 

0.0 

7.8 

8.7 

6.9 

23.7 

42.6 

63.3 

52.4 

53.2 

7.9 

42.6 

58.3 

42.3 

46.1 

15.8 

5.6 

17.2 

16.8 

15.4 

15.8 

0.0 

8.9 

11.4 

9.3 

10.5 

0.0 

5.0 

4.0 

4.5 

2.6 

0.0 

7.2 

9.4 

6.7 

2.6 

0.0 

4.4 

10.1 

5.7 

2.6 

0.0 

2.2 

9.4 

4.5 

68.4 

25.9 

46.7 

65.1 

52.5 

21.1 

9.3 

13.3 

32.2 

20.2 

21.1 

3.7 

3.9 

7.4 

6.7 

5.3 

1.9 

2.8 

8.1 

4.8 

5.3 

3.7 

6.1 

18.1 

10.0 

0.0 

0.0 

0.0 

4.7 

1.7 

15.8 

13.0 

12.8 

37.6 

21.9 

15.8 

9.3 

18.3 

26.9 

20.0 

29.0 

7.4 

7.2 

22.8 

14.7 

42.1 

46.3 

26.1 

72.5 

46.6 

18.4 

1.9 

0.0 

9.4 

5.2 

13.2 

0.0 

0.0 

28.9 

11.4 

18.4 

35.2 

0.0 

7.4 

8.8 

10.5 

13.0 

26.1 

63.8 

36.3 


Sample size 38 54 180 149 421 

(continued) 
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Table 2.8 (continued) 


SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by two 
sites and in program tracking systems by two sites. 

NOTES: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 
Provider staff members documented all phone and in-person advancement coaching meetings with participants 
during that time period. 

WorkAdvance group members may attend more than one meeting during the data collection period. 

Some statistics may not add to 100 percent due to missing values. 

a Only one meeting per person and day is counted. Two participants appear to have met with staff members 
twice on the same day. 


WorkAdvance staff members reported that the ways they saw WorkAdvance partici- 
pants advancing most quickly or commonly often differed by sector, but staff members held 
different opinions, even regarding the same sector, about whether changing jobs or advocating 
for better pay at one’s current employer was the most expeditious route to advancement. At the 
same time, they noted a number of services that they provided as being effective ways to help 
people advance. These included motivational coaching, teaching negotiation tactics, instructing 
participants to log their on-the-job accomplishments to formally demonstrate their value to their 
employer, and providing general encouragement and moral support. 

When asked if some types of WorkAdvance participants were more likely to advance 
than others, WorkAdvance staff members emphasized the role of perceived attitudes. As exam- 
ples, they indicated that WorkAdvance participants who took ownership of their goals, believed 
in themselves, were committed to attending training, and communicated more frequently with 
staff members were more likely to advance in their jobs. In contrast, WorkAdvance participants 
who were either overconfident or lacked self-confidence were viewed as likely to have trouble 
advancing. In addition, staff members thought that participants with mental health problems or 
criminal backgrounds might have insunnountable challenges to advancing. 


Changes in Components and Participation Patterns over Time 

• WorkAdvance providers adjusted the delivery of various program com- 
ponents over time, as staff members learned what seemed to work best 
in practice. 

The WorkAdvance programs did, in general, mature over time — that is, the providers 
changed the delivery of some components as they gained operational experience, just as they 
adjusted the content of their career readiness and occupational skills training classes. For exam- 
ple, Per Scholas and St. Nicks Alliance initially did not have designated career coaches to lead 


57 



The WorkAdvance Study 
Table 2.9 


Content of Advancement Coaching Meetings for WorkAdvance Group Members 
Who Attended a Meeting While Employed, Cross-Site 





Madison 




Per 

St. Nicks 

Strategies 

Towards 


Activity (%) 

Scholas 

Alliance 

Group 

Employment 

Overall 

Among WorkAdvance group members who attended a meeting 





Ever engaged in an advancement coaching meeting 






while employed 

68.4 

70.4 

79.4 

75.8 

76.0 

Average number of advancement coaching meetings 






that took place while the participant was: a 

Employed 

0.7 

1.1 

1.3 

1.8 

1.4 

Not employed 

0.1 

0.6 

0.5 

1.1 

0.7 

Sample size 

38 

54 

180 

149 

421 

Among WorkAdvance group members employed at time of meeting 





Discussed job search-related topic 

38.5 

34.2 

40.6 

59.3 

46.3 

Discussed retention-related topic 

34.6 

60.5 

74.8 

68.1 

67.5 

Discussed advancement-related topic 

65.4 

36.8 

55.2 

79.7 

62.5 

Discussed other topic 

50.0 

52.6 

26.6 

69.0 

46.6 

Sample size 

26 

38 

143 

113 

320 


SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by two 
sites and in program tracking systems by two sites. 


NOTES: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 
Provider staff members documented all phone and in-person advancement coaching meetings with participants 
during that time period. 

WorkAdvance group members may attend more than one meeting during the data collection period. 

Some statistics may not add to 100 percent due to missing values. 

“Only one meeting per person and day is counted. Two participants appear to have met with staff members 
twice on the same day. 

pre- and postemployment advancement coaching; preemployment coaching fell primarily to the 
career readiness instructors. After about a year of program operations, both providers hired a 
designated career coach to deliver pre- and postemployment advancement coaching. Then, in 
fall 2013, Per Scholas broadened the roles of its career readiness instructors to include career 
coaching with a focus on advancement. As another example, Towards Employment initially had 
participants start their interaction with the program by meeting with a case manager, who con- 
ducted a needs assessment focused on participants’ barriers to employment; the organization 
eventually began to use a different kind of assessment tool that emphasized goal-setting rather 
than barriers. 
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• A “mature” and “fully implemented” WorkAdvance program was more 
likely to have been experienced by individuals who enrolled in the pro- 
gram later in the random assignment period (the “late cohort”) than by 
individuals who enrolled in the program earlier (the “early cohort”). 

Notably, many WorkAdvance group members had already made their way through a 
significant part of the WorkAdvance model before mid-2013. As a result, coaching truly fo- 
cused on advancement was more likely to have been experienced by individuals who enrolled 
in WorkAdvance in the late cohort than by individuals who enrolled earlier. 

While theoretically maturation might also be indicated by changes in participation pat- 
terns for later program enrollees compared with early program enrollees — if program staff 
members were better able to engage and support participants in their activities as the staff 
learned, through practice, what techniques work best operationally — in reality participation 
patterns did not differ much between early and late enrollees within each site. As Table 2.10 
shows, skills training participation and completion rates were high for both the early and the late 
cohorts at Per Scholas and St. Nicks Alliance (providers that consistently operated “training 
first” WorkAdvance models). Rates of starting and completing skills training did change from 
the early to the late cohort at Madison Strategies Group and Towards Employment, but this 
change primarily reflects those providers’ initial two-track approach to WorkAdvance, which 
allowed some participants to be placed directly into jobs rather than engaging in training first. 
Once those providers began to direct more participants to training before employment, their 
training start and completion rates became higher and much closer to those of the other two 
providers. This suggests that program “maturation” was probably evident more in the quality of 
the services provided or in the smoothness of program operations than in the likelihood of indi- 
viduals receiving specific services. 


Conclusion 

Key differences in implementation across the WorkAdvance providers were apparent and cen- 
tered on a number of factors. First, the providers had different starting points. Per Scholas had a 
sizable head start, in its delivery of sector training in information technology for more than 10 
years before WorkAdvance and in its previous participation in a random assignment research 
study. Second, Towards Employment’s operational complexity — stemming from its adminis- 
trative structure, which required collaboration across a number of institutions to deliver services 
in two completely distinctive sector programs, one of which was in a sector relatively new to it 
— set it apart from the other providers. Third, two sites (Madison Strategies Group and To- 
wards Employment) initially operated a dual-track approach to WorkAdvance, which resulted 
in early enrollees at those two sites being less likely than enrollees at the other two sites to ever 
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The WorkAdvance Study 
Table 2.10 


Indicators of Participation in Program Group Activities 
Within 18 Months After Random Assignment, 
by Random Assignment Cohort, Cross-Site 


Participation in program activity (%) 

Per 

Scholas 

St. Nicks 

Alliance 

Madison Towards 

Strategies Group Employment 

Overall 

Among enrollees in the early cohort 

Ever participated in any career readiness activity 11 

95.2 

83.5 

99.4 

98.8 

95.0 

Ever started skills training 

95.2 

83.5 

51.5 

53.6 

70.8 

Ever completed skills training 

79.4 

79.5 

39.3 

42.3 

59.4 

Ever obtained a credential in targeted sector 13 

70.9 

76.4 

38.7 

26. 8 C 

52.2 

Sample size 

189 

127 

173 

168 

657 

Among enrollees in the late cohort 

Ever participated in any career readiness activity' 1 

96.3 

81.7 

97.2 

97.8 

94.3 

Ever started skills training 

96.3 

81.7 

72.2 

76.2 

81.1 

Ever completed skills training 

77.5 

78.3 

57.8 

57.5 

66.4 

Ever obtained a credential in targeted sector 3 

74.4 

70.4 

58.3 

36.5° 

58.3 

Sample size 

160 

115 

180 

181 

636 


SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 
Strategies Group (MSG), and Towards Employment (TE). 


NOTES: The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late 
cohort includes all sample members randomly assigned in or after Quarter 4, 2012. 

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. 
a The first program activity at PS and SNA is skills training, which is offered in combination with career 
readiness training and includes help with developing a career plan, resumes, and interview preparation. At MSG, 
the first progam activity is career readiness training. At TE, the first program activity is an initial assessment 
wherein career goals and barriers to employment are discussed. 

b Credentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 
reporting of locally recognized credentials obtained in the targeted sector. 

c TE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 
obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 
having obtained a credential. 
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start and complete occupational skills training. Fourth, the providers differed in their delivery of 
career readiness and postemployment services. Per Scholas, by virtue of having all its training 
on-site and therefore having more access to program participants, delivered more frequent ca- 
reer readiness services, and Madison Strategies Group took an innovative incentive approach to 
the delivery of postemployment services. Fifth, Per Scholas took an approach to postemploy- 
ment services that did not quite follow the model: One-on-one, individualized postemployment 
career advancement coaching was less frequent than called for by the model and received less 
emphasis than did group activities focused on advancement, such as the advanced certification 
classes for alumni, which were starting in earnest around the time that field research for the 
evaluation was wrapping up. Finally, the nature of the different sectors targeted by each provid- 
er influenced the providers’ emphasis on how participants could best advance. In the infor- 
mation technology and health care sectors, more emphasis was generally placed on working in a 
professional environment and staying on the job longer; in the manufacturing and environmen- 
tal remediation sectors, more emphasis was placed on refining skills on the job and taking on 
additional responsibilities that could be made use of during subsequent job searches or employ- 
er negotiations. 

Most of this chapter examined the implementation of WorkAdvance in light of the de- 
gree to which the program’s components were advancement focused and employer driven. Im- 
plementation of the WorkAdvance model also can be examined in light of similarities to or dif- 
ferences from the features of the programs in the Sectoral Employment Impact Study (SEIS). 
The researchers in that study suggested that the features that made those programs successful 
were “a good understanding of and connection to industry needs, careful screening to identify 
appropriate clients, a sector-focused approach to training, individualized support services and 
the organizational capacity to put all of these ingredients together .” 24 Many of these features 
were exhibited by the WorkAdvance providers, though not all of them and not equally across 
the providers. Most important, perhaps, is that the organizational capacities of the provider or- 
ganizations varied — from each other, and also from those in the SEIS. To be selected for the 
SEIS, programs had to be operating for at least three years and were asked to simply continue 
running the programs they already were operating, as opposed to implementing features of a 
specific model. A key difference between the two sets of programs, therefore — with the excep- 
tion of Per Scholas — was their experience operating sector programs and the degree to which 
they had deep connections with employers and intermediaries in the sector. These connections 
take time to develop, even for an organization with experience in the sector if it establishes itself 
in a new location, such as Madison Strategies Group did in Tulsa. 

The next chapter continues a discussion of participation in WorkAdvance but focuses 
on the treatment differential between the WorkAdvance group members and control group 


24 Maguire et al. (2010), p. 48. 
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members. Information on participation in the next chapter also comes from a different data 
source — it is based on survey data, which captures training and other activities that occurred 
within WorkAdvance, as well as outside of WorkAdvance, and covers a slightly longer follow- 
up period. 
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Chapter 3 

WorkAdvance-Control Group Differences in 
Services Received 


Chapter Highlights 

• Across all sites, WorkAdvance had high participation in services, producing 
large impacts in all the key service domains despite high control group ser- 
vice receipt levels. 

• WorkAdvance participants completed training and earned credentials at a 
much higher rate than control group members. 

• Impacts were particularly large for postemployment services — from about 
40 to 70 percentage points, depending on the site. 


Introduction 

The previous chapter described the implementation of the WorkAdvance program by each of 
the providers and indicated the percentage of WorkAdvance group members who, according to 
the programs’ management infonnation system (MIS) data, participated in WorkAdvance ser- 
vices within the first 1 8 months of enrollment. The data shown in the previous chapter indicate 
that participation in employment services and training among WorkAdvance group members 
was high, but how much higher was it relative to participation among control group members? 
This chapter focuses on the differences in services received by the WorkAdvance group and the 
control group at each of the four sites. Using data from the Year 2 Survey, which was adminis- 
tered to both groups, this chapter examines the extent to which WorkAdvance increased partici- 
pation, compared with what would have happened in the absence of the program, in the follow- 
ing types of activities: 

• career advice and advancement help 

• education and training 

• job search assistance 

• postemployment services 
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The Year 2 Survey measures participation over the first 18 to 30 months after random 
assignment. It asked respondents whether they participated in employment-related services and 
education and training activities and, if so, the duration, type, and provider of the service. 1 

Comparing the Year 2 Survey responses regarding participation by the WorkAdvance 
group members and by the control group members reveals WorkAdvance’s estimated “impacts” 
on service receipt and training participation. (Box 3.1 explains how to read the impact exhibits 
in this report.) In other words, these results measure the “treatment contrast,” that is, the differ- 
ences in service receipt that were produced by WorkAdvance. The underlying hypothesis of the 
WorkAdvance model is that helping low-income adults improve both their career readiness and 
their technical skills, through sector-focused training and close relationships with employers, 
will substantially enhance their opportunities for long-term employment. Together with the 
findings from Chapter 2, which discussed the extent to which WorkAdvance services were fo- 
cused on advancement and reflective of employer needs, knowledge of the results presented in 
this chapter is central to understanding the employment and other economic impacts of Work- 
Advance that are presented in Chapter 5. 

Following a summary of WorkAdvance’s participation impacts, the chapter discusses 
findings for each of the abovementioned service areas in the order of the WorkAdvance pro- 
gram flow. Thereafter, participation impacts by time of enrollment are presented to examine 
whether programmatic changes implemented during and after the fall of 2012 (discussed in 
Chapter 2) made a difference in the size of the participation impacts. Finally, the chapter ends 
with key points about variation in these impacts by site and time of enrollment. 


Summary of Participation Impacts 

WorkAdvance produced substantial increases in training and employment services at all four 
sites. This is illustrated in Figure 3.1, which presents program and control group rates of partici- 
pation in the core services that the WorkAdvance providers were expected to deliver. These 
impacts are noteworthy given the large proportions of individuals who would have participated 
in similar services without WorkAdvance — as indicated by the proportions of control group 
members who reported engaging in training and employment services. (Box 3.2 presents more 
information on the services received by control group members.) 2 


1 As discussed in detail in Appendix A, the overall response rate for the Year 2 Survey was about 80 per- 
cent. This rate did not differ significantly between WorkAdvance group members and control group members. 

2 To help determine, near the start of the WorkAdvance evaluation, whether there was a sufficient differ- 
ence between the services received by the program group and by the control group, a Control Services Survey 
(CSS) was fielded to a very small sample of early enrollees about six months after they entered the study. 
While statistically significant differences between the groups in the take-up of employment- and training- 
related services were found in this early check, the CSS clearly showed that some control group members were 
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Box 3.1 

How to Read the Tables in This Report 


Most tables in this report use a similar format, illustrated below. The table shows key par- 
ticipation outcomes for the WorkAdvance and control group Year 2 Survey respondents at 
the Per Scholas site — whether they obtained an employment service and what type. (The 
survey question was designed to capture both WorkAdvance services for program group 
members and services from any employment counselors or programs that might be availa- 
ble to members of either research group.) For example, the table shows that 93 percent of 
the site’s WorkAdvance respondents reported obtaining an employment service, compared 
with 65 percent of control group respondents. 

Because study participants were assigned randomly to either the WorkAdvance group or 
the control group, the effects of WorkAdvance can be estimated by the difference in out- 
comes between the two groups. The “Difference” column in the table shows the WorkAd- 
vance group’s rate of service receipt minus the control group’s rate — that is, the pro- 
gram’s impact on service receipt at that site. For example, the impact on obtaining an em- 
ployment service is calculated by subtracting 65 from 93, yielding 28 percentage points. 

Differences marked with asterisks are statistically significant, meaning that it is quite un- 
likely that the differences arose by chance. The number of asterisks indicates whether the 
impact is statistically significant at the 1 percent, 5 percent, or 1 0 percent level (the lower 
the level, the less likely that the impact is due to chance). For example, as shown below, 
WorkAdvance had a statistically significant impact of 28 percentage points on the propor- 
tion of participants who reported obtaining an employment service. Three asterisks indi- 
cate that this impact is statistically significant at the 1 percent level. 


Summary of Year 2 Impacts on Participation in 
Employment Services and Training, Per Scholas 


Outcome (%) 

WA 

C 

Difference 

(Impact) 


Obtained an employment service 

93.3 

65.4 

27.9 

*** 

Career readiness service 

89.2 

60.3 

28.9 

*** 

Job search service 

81.7 

36.2 

45.5 

*** 

Postemployment service 

64.3 

25.5 

38.9 

*** 


NOTE: WA = WorkAdvance group; C = control group. 
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Figure 3.1 

Summary of Year 2 Impacts on Participation in Employment Services and Training, by Site 


Career advice and advancement services 3 Started skills training 





Per Scholas St. Nicks Madison Towards Per Scholas St. Nicks Madison Towards 

Alliance Strategies Group Employment Alliance Strategies Group Employment 





Per Scholas St. Nicks Madison Towards Per Scholas St. Nicks Madison Towards 

Alliance Strategies Group Employment Alliance Strategies Group Employment 
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Figure 3.1 (continued) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Sample sizes may vary because of missing values. 

a This measure includes career planning, job readiness, and services related to keeping or 
advancing on a job. 

b This measure includes finding a specific job lead in the target field, finding a specific job 
opportunity to apply for, and scheduling a job interview. 

c Postemployment services include those received while an individual is employed: Respondent 
obtained help keeping current or most recent job, or employment service providers contacted the 
employer about how the respondent could keep the job or advance. 


Box 3.2 

To What Extent and Where Did Control Group Members Receive 
WorkAdvance-Like Services? 

How many control group respondents received services that were the same as or similar to 
the ones that WorkAdvance providers offered to WorkAdvance group members? And who 
offered these services? The table below helps to answer these questions. It provides site- 
by-site control group participation rates in employment and training services by provider. 
Overall, between 64 percent and 76 percent of control group members reported in the 
Y ear 2 Survey that they received employment and occupational skills training services in 
their communities. 

• Between 5 1 percent and 68 percent of control group members, depending on the site, 
received an “employment service”: that is, help preparing for a career, help searching 
for a job, or postemployment assistance, from an employment or government agency 
or a community-based organization. 

• Between 13 percent and 21 percent of control group members participated in occupa- 
tional skills training in the targeted sector. Though control group members found their 
way to a broad range of training providers in the targeted sectors, community colleges 
and for-profit training providers were the most common ones. At each site, the most 
common types of framing providers used by control group members were roughly the 
same for occupational skills training in and outside the targeted sector. 

• A special analysis of verbatim responses to a survey question about the type of occu- 
pational skills training programs that control group members attended showed some 
site-level variation. The most common nontargeted sectors in which control group 
members sought skills training were health care, transportation, and security (Per 
Scholas); health care and transportation (St. Nicks Alliance); health care and security 
(Madison Strategies Group); and customer service, cosmetology, and hospitality (To- 
wards Employment). 

(continued) 
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Box 3.2 (continued) 

Control Group Participation Rates in Employment Services and 
Occupational Skills Training, by Site 



Per 

St. Nicks 

Madison 

Strategies 

Towards 

Outcome (%) 

Scholas 

Alliance 

Group Employment 

Ever received any employment service 

or participated in skills training 11 

76.2 

76.1 

63.9 


68.5 

Ever received any employment service through an 

employment or government agency or CBO 

65.4 

67.8 

50.7 


58.7 

Ever participated in skills training 

46.8 

45.2 

30.0 


38.5 

In targeted sector 

20.3 

21.3 

13.3 


15.3 

Community/2-year/technical college 

6.7 

2.9 

9.1 


8.4 

For-profit training provider 

1.1 

0.5 

0.0 


0.0 

High school/community school 

7.0 

9.6 

2.3 


3.9 

One-Stop Career Center/government agency 

2.8 

3.6 

0.0 


0.8 

Other c 

3.4 

4.8 

2.1 


2.9 

In other sector 1 ' 

18.0 

18.7 

12.3 


13.5 

Community/2-year/technical college 

5.6 

5.7 

7.4 


5.2 

For-profit training provider 

2.0 

0.0 

0.0 


0.7 

High school/community school 

4.2 

7.0 

1.7 


1.8 

One-Stop Career Center/government agency 

1.6 

0.6 

0.4 


1.2 

Other c 

5.5 

8.5 

2.8 


5.9 

Sample size (total = 983) 

265 

179 

263 


276 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: Sample sizes may vary because of missing values. Respondents were allowed to report 
on up to five occupational skills training programs. 

“This outcome includes job-specific skills training, job search assistance, career readiness 
assistance, and postemployment assistance. 

b Respondents were asked to report on employment services received from an employment 
agency, government agency, or community-based organization (CBO). They were not 
asked to specify where the service was provided. 

c Other occupational skills training providers include nonprofit organizations, four-year 
colleges, the respondent's workplace, and nonspecified providers. 

d This outcome includes participation in any occupational skills training programs geared 
toward occupations in sectors other than those targeted by the WorkAdvance provider. 
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The fact that so many control group members found and participated in these services 
indicates that WorkAdvance had a high bar to meet and exceed, in terms of delivering “addi- 
tional” employment and training services to WorkAdvance group members. 

• Across all sites, WorkAdvance resulted in very large increases in partic- 
ipation in every category of services, compared with what would have 
happened in the absence of the program. 

Figure 3.1 shows that despite the high control group participation levels, WorkAdvance 
produced large increases in service receipt. The estimated impacts on the receipt of career read- 
iness services ranged from 26 percentage points at St. Nicks Alliance to 46 percentage points at 
Madison Strategies Group. Similarly, the increases in the receipt of job search services ranged 
from 31 percentage points at St. Nicks Alliance to 52 percentage points at Madison Strategies 
Group. WorkAdvance also increased the receipt of postemployment services, which was the 
last of all core service components to be fully developed, by roughly 40 percentage points or 
more at every site. Notably, Madison Strategies Group yielded a 72 percentage point increase in 
receipt of this type of service, with a take-up rate of 14 percent for the control group and an im- 
pressive 86 percent for the WorkAdvance group. 

WorkAdvance increased participation in occupational skills training (vocational train- 
ing) by 30 percentage points or more at every site, and increased the occupational skills training 
completion rate by 27 percentage points or more. (Impacts are shown in the following sections.) 
The impacts on participating in training in the sectors targeted by the WorkAdvance programs 
are especially pronounced, ranging from 38 to 49 percentage points by site. 

The sections that follow review site-by-site impacts on participation and related out- 
comes, by WorkAdvance model component. The findings demonstrate that the WorkAdvance 
model received a good test, in terms of providing a much higher amount of employment- and 
training-related help than would have been received in the absence of the program. 


Impacts on Participation in Career Readiness Services, by Site 

As discussed in Chapter 2, as well as in the previous WorkAdvance report, 3 the career advice 
and advancement component, referred to as “career readiness,” covers a wide range of services, 
including career planning, job readiness, and preemployment job retention and advancement- 
related help. 

• WorkAdvance group members at all four sites were much more likely 
than control group members to report receiving career readiness ser- 


3 Tessler et al. (2014). 


69 



vices, with the impacts ranging from about 29 percentage points for Per 
Scholas to 46 percentage points at Madison Strategies Group. 

Table 3.1 captures the effects of WorkAdvance on the receipt of career readiness help. 
The Year 2 Survey included multiple questions about the nature of career readiness help that 
study participants received from staff members of various types of organizations — employ- 
ment, government, and community-based — regardless of whether this help was offered as part 
of WorkAdvance. While the survey did not inquire about the frequency or quality of the career 
readiness help received, respondents were asked to assess the overall importance of any type of 
employment service they received. 

Developing a career plan is one of the initial activities built into the WorkAdvance 
model, and each of the providers made it a priority to offer this help in the earliest stage of ser- 
vice delivery. Therefore, it is not surprising that WorkAdvance group members were much 
more likely than control group members to report that they received help in making a career 
plan. Impacts in reporting receiving such help ranged from 34 percentage points for St. Nicks 
Alliance to 55 percentage points for Madison Strategies Group. The relatively low incidence of 
receiving help in this area among control group members indicates that this population does not 
ordinarily receive these types of services in the community at large; this is especially true at the 
Madison Strategies Group site, where only 1 1 percent of control group members reported re- 
ceiving help making a career plan. 

Similarly, as described in the previous chapter, job readiness training was also offered 
in the early phase of WorkAdvance service delivery, and therefore participation levels for the 
WorkAdvance group were expected to be high. Between 70 percent (for Towards Employment) 
and 82 percent (for Per Scholas) of WorkAdvance group members reported receiving help ob- 
taining training on how to be a good or better employee. These rates are much higher than the 
respective control group rates, yielding statistically significant increases in the receipt of this 
type of help, ranging from 39 percentage points at St. Nicks Alliance (where overall participa- 
tion levels in career readiness activities among WorkAdvance group members were somewhat 
lower than at the other providers) to 61 percentage points at Madison Strategies Group (which 
was the only provider to enroll WorkAdvance group members in this type of training immedi- 
ately following entry into the program). 

• Considering the three types of career readiness services measured via 
the Year 2 Survey — help with career planning, help with job readiness, 
and help keeping or advancing in a job — WorkAdvance produced the 
largest increases in providing individuals with help keeping or advanc- 
ing in a job (Table 3.1). 
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The WorkAdvance Study 
Table 3.1 

Year 2 Impacts on Participation in Career Readiness Services, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 





Difference 



Difference 



Difference 



Difference 

Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Received a career readiness service 3 

89.2 

60.3 

28.9 

*** 

90.3 

64.2 

26.2 

*** 

90.1 

43.8 

46.3 

*** 

87.2 

52.2 

35.0 

*** 

Employment services in which respondent participated 















Career planning 

76.0 

44.9 

31.1 

*** 

75.6 

48.4 

27.2 

*** 

81.6 

28.1 

53.5 

*** 

76.0 

37.0 

39.0 

*** 

Finding a suitable career 

61.0 

33.4 

27.6 

*** 

51.9 

27.4 

24.5 

*** 

61.5 

18.3 

43.3 

*** 

61.5 

26.2 

35.3 

*** 

Making a career plan 

60.7 

26.6 

34.0 

*** 

58.1 

24.3 

33.8 

*** 

66.2 

11.4 

54.9 

*** 

65.3 

20.5 

44.8 

*** 

Finding skills training 

66.2 

35.5 

30.7 

*** 

62.2 

42.2 

20.0 

*** 

69.7 

18.2 

51.5 

*** 

68.9 

26.5 

42.4 

*** 

Job readiness 3 

88.0 

53.7 

34.3 

*** 

84.0 

55.2 

28.9 

*** 

86.7 

38.0 

48.8 

*** 

83.5 

43.4 

40.1 

*** 

Preparing a resume 

79.5 

39.2 

40.2 

*** 

67.8 

45.7 

22.1 

*** 

78.1 

28.6 

49.5 

*** 

73.9 

34.8 

39.1 

*** 

Completing a job application 

49.2 

20.7 

28.5 

*** 

47.9 

19.7 

28.2 

*** 

57.7 

11.4 

46.3 

*** 

53.9 

19.5 

34.4 

*** 

Appearing professional 

78.0 

33.3 

44.8 

*** 

65.7 

31.9 

33.8 

*** 

76.1 

16.1 

60.0 

*** 

71.5 

27.8 

43.7 

*** 

Enrolling in job readiness training 

82.2 

29.9 

52.3 

*** 

71.8 

32.5 

39.2 

*** 

76.7 

15.7 

61.1 

*** 

70.1 

26.2 

43.9 

*** 

Getting a job recommendation 

65.8 

27.4 

38.4 

*** 

57.5 

22.9 

34.6 

*** 

53.9 

13.6 

40.3 

*** 

55.7 

20.6 

35.0 

*** 

Practicing for a job interview 

80.5 

29.4 

51.1 

*** 

70.3 

30.5 

39.7 

*** 

76.6 

11.3 

65.3 

*** 

70.0 

27.6 

42.5 

*** 

Keeping or advancing on a job d 

78.8 

30.2 

48.6 

*** 

68.5 

34.3 

34.2 

*** 

72.3 

15.9 

56.5 

*** 

70.7 

29.0 

41.6 

*** 

Advice on keeping a job 

75.1 

24.4 

50.7 

*** 

62.6 

28.6 

34.0 

*** 

67.5 

13.5 

54.0 

*** 

65.6 

23.6 

42.0 

*** 

Managing problems with 
coworkers 

71.8 

26.1 

45.7 

*** 

62.1 

27.7 

34.3 

*** 

68.2 

12.8 

55.5 

*** 

66.8 

26.0 

40.9 

*** 


(continued) 




Table 3.1 (continued) 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 

Difference Difference Difference Difference 

Outcome (%) WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) 


Respondent employed since RA and discussed: 


Passing probation at new job 

28.3 

7.0 

21.3 

*** 

21.7 

7.8 

13.9 

*** 

32.9 

4.7 

28.3 

*** 

26.5 

6.6 

19.9 

*** 

Obtaining an employee handbook 

27.3 

9.7 

17.6 

*** 

25.0 

12.9 

12.1 

*** 

43.2 

10.0 

33.2 

*** 

30.0 

10.7 

19.3 

*** 

Understanding job duties 

41.1 

15.5 

25.6 

*** 

34.7 

12.8 

21.8 

*** 

50.7 

12.8 

37.9 

*** 

39.7 

12.8 

26.9 

*** 

Understanding how to advance 

Making child care or transportation 

42.2 

13.8 

28.4 

*** 

33.2 

15.2 

18.0 

*** 

51.6 

10.3 

41.2 

*** 

41.2 

14.2 

27.0 

*** 

plans 

21.9 

7.7 

14.1 

*** 

27.9 

9.0 

18.9 


37.9 

5.3 

32.7 

*** 

40.1 

12.4 

27.7 

*** 

Respondent said contact with staff is 
important for making progress 
toward career goals 

75.2 

43.9 

31.3 

*** 

80.7 

54.8 

25.9 

*** 

84.2 

44.9 

39.3 

*** 

82.4 

59.0 

23.4 

*** 

Sample size (total = 2,058) 

287 

265 



205 

179 



297 

263 



286 

276 




SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

a Category includes career planning, job readiness, and services related to keeping or advancing on a job (as defined in footnotes b through d). 
b Category includes identifying skills and interest in a suitable career, making a career plan, and advice on getting training. 

■Category includes preparing a resume, filling out a job application, help presenting oneself professionally, obtaining training on how to be a good or better 
employee, getting a recommendation, practicing for a job interview. 

Category includes services related to keeping or advancing in a job that might be received before a job is actually obtained. 



The impacts of WorkAdvance on help with keeping or advancing in a job ranged from 
34 percentage points (St. Nicks Alliance) to 57 percentage points (Madison Strategies Group). 
Control group rates of help received in this employment service area are similar across sites, 
except at the Madison Strategies Group site, where control group receipt rates were lower. The 
size of these increases is driven mostly by high receipt rates among WorkAdvance group mem- 
bers, as opposed to low receipt rates among control group members. For example, the propor- 
tion of WorkAdvance group respondents reporting that they received help with keeping or ad- 
vancing in a job is notably high at Per Scholas, where approximately 79 percent reported that 
they obtained either advice on keeping a job or help managing problems with coworkers. One 
factor that may help explain the high receipt rate at this site is that these topics were covered at 
the same time that Per Scholas provided its occupational skills training, as discussed in Chapter 
2. WorkAdvance also produced large and positive impacts, across all sites, on the receipt of 
other advancement-related help, especially with regard to having discussions with staff mem- 
bers about how to advance in a job. 

WorkAdvance also appeared to increase people’s perception that it is important to get 
help with career readiness. At all sites there were very large impacts on the proportion of sample 
members who said that contact with staff members was important to making progress on career 
goals, especially at Per Scholas and Madison Strategies Group, where increases above the con- 
trol group exceeded 30 percentage points. As noted in Chapter 2, these two providers had strong 
employer relationships and conducted individualized job matching between participants and 
employers. 


Impacts on Participation in Education and Training, by Site 

The impacts of WorkAdvance on education and training can be examined from multiple angles. 
First, Table 3.2 presents the impacts of WorkAdvance on participation in any type of education 
and training. This includes measures of take-up, persistence, length, type, completion, and cre- 
dential acquisition. Next, Figure 3.2 illustrates the percentage of WorkAdvance and control 
group members who participated in any occupational skills training program in each month fol- 
lowing random assignment. Finally, Table 3.3 focuses on the effects of the program on partici- 
pation in a primary occupational skills training program. This primary training table also covers 
any differences between the two groups in how sample members assessed these training pro- 
grams and in the amounts they paid for the training. 

• WorkAdvance significantly increased participation in education and 
training, most notably in the form of occupational skills training, at all 
four sites (Table 3.2). 
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The WorkAdvance Study 
Table 3.2 

Year 2 Impacts on Participation in Education and Training, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 


Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Participated in education or training 

84.4 

63.0 

2i 4 *** 

81.2 

60.6 

20.6 *** 

76.1 

49.5 

26.6 *** 

80.1 

54.5 

25.6 *** 

Occupational skills training 1 ' 

78.4 

46.8 

31.6 *** 

75.1 

45.2 

29 9 *** 

66.8 

30.0 

36.8 *** 

69.9 

38.5 

32 4 *** 

In targeted sector 

69.4 

20.3 

49 1 *** 

60.0 

21.3 

3g y *** 

55.3 

13.3 

42 j *** 

61.1 

15.3 

45.8 *** 

College courses 

27.5 

28.7 

- 1.2 

14.7 

15.8 

- 1.0 

17.6 

18.9 

-1.3 

23.2 

22.6 

0.6 

ABE/GED/HS diploma preparation 

2.7 

2.0 

0.8 

7.3 

6.2 

1.0 

5.4 

5.3 

0.1 

5.3 

7.5 

- 2.2 

Short-term classes (1-5 days) 

35.0 

23.9 

2 2 j *** 

36.5 

27.7 

8.7 * 

30.3 

24.9 

5.4 

42.5 

25.1 

2 y 5 *** 

Completed any education or training 

66.9 

36.1 

30.8 *** 

63.7 

42.7 

21.0 *** 

47.8 

25.2 

22.6 *** 

55.2 

28.8 

26.4 *** 

Obtained a degree or credential b 

66.2 

30.1 

36.1 *** 

62.1 

37.8 

24 3 *** 

49.4 

21.6 

22 g *** 

51.2 

25.5 

25 y *** 

Average number of skills training 

programs started 

1.1 

0.5 

0.6 *** 

1.0 

0.6 

Q 4 *** 

0.8 

0.3 

q 4 *** 

1.0 

0.4 

0.6 *** 

Completed a skills training 

program 

62.0 

25.3 

36.8 *** 

59.9 

32.5 

22 4 *** 

43.0 

16.4 

26.6 *** 

49.6 

18.8 

30 9 *** 

Dropped out of a skills training 

program 

7.9 

5.1 

2.8 

6.3 

1.3 

5.0 ** 

9.6 

5.2 

4.4 * 

9.0 

3.1 

3 9 *** 

Average number of months in skills 

training^ 

4.3 

2.1 

2 2 *** 

3.2 

1.8 

Y 4 *** 

3.7 

1.0 

2.6 *** 

4.1 

1.5 

2.6 *** 

Among those who started 

5.9 

5.9 

0.0 

4.9 

5.4 

-0.5 

6.2 

4.4 

1.7 

6.6 

5.6 

1.0 

Obtained a credential 

58.1 

19.0 

39 y *** 

55.5 

26.0 

29 5 *** 

36.3 

13.7 

22.6 *** 

39.9 

15.0 

24 9 *** 

In targeted sector 

54.1 

7.9 

45 \ *** 

48.3 

17.6 

3Q 2 *** 

32.1 

6.9 

25.2 *** 

36.5 

9.6 

26.9 *** 
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Table 3.2 (continued) 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Occupational skills training provider 
WorkAdvance provider 

68.0 

0.0 

68.3 *** 

54.6 

1.7 

53.0 *** 

49.9 

0.1 

49 9 *** 

54.8 

1.6 

53.2 *** 

Community/2-year/technical college 

4.3 

12.1 

J~J g *** 

6.8 

9.2 

-2.5 

12.0 

16.0 

-4.0 

9.8 

13.6 

-3.7 

4-year college 

0.3 

3.1 

- 2.8 ** 

0.1 

0.5 

-0.4 

0.3 

0.4 

- 0.1 

0.0 

0.7 

-0.7 

For-profit training provider 

1.7 

10.6 

_g g *** 

7.8 

15.4 

-7.6 ** 

2.6 

4.3 

-1.7 

6.6 

5.6 

0.9 

HS/community school 

0.4 

3.4 

-3.0 ** 

0.9 

1.8 

- 0.8 

1.3 

1.6 

-0.4 

1.7 

1.6 

0.1 

One-Stop Career Center/ unemployment 
office/government agency 

0.7 

0.4 

0.3 

3.3 

0.8 

2.5 * 

0.0 

0.4 

-0.4 

0.0 

0.4 

-0.4 

Nonprofit organization/church 

0.9 

4.4 

-3.5 ** 

3.2 

4.3 

- 1.0 

0.3 

0.4 

- 0.1 

4.1 

2.0 

2.1 

Workplace 

1.7 

1.5 

0.2 

3.0 

5.7 

-2.7 

1.1 

2.3 

- 1.2 

2.9 

2.1 

0.8 

Other 

2.4 

3.5 

- 1.1 

4.4 

4.0 

0.4 

0.6 

0.5 

0.2 

1.9 

2.4 

-0.5 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; ABE = adult basic education; GED = general educational development; HS = high school. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

Italic type indicates that the measure is nonexperimental; thus, statistical tests were not performed. 

“Includes all reports of occupational skills training from sections A, D, and E of the survey. All subsequent occupational skills training measures include 
only skills training programs recorded in section E. For example, 113 people reported participating in skills training in the survey but not in section E. 

b Measure includes completing high school; obtaining a GED certificate; obtaining an associate's, bachelor's, or professional degree; and obtaining a 
credential as part of a skills training program. 

“Measure includes only participation in occupational skills training within 18 months following each respondent's date of random assignment (the common 
follow-up period). 
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The WorkAdvance Study 
Figure 3.2 


Percentage of WorkAdvance and Control Group Members Participating in Occupational Skills Training, 

by Month Relative to Random Assignment and Site 

Per Scholas 



Month relative to random assignment 


St. Nicks Alliance 


WA group 
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Figure 3.2 (continued) 


Madison Strategies Group 


WA group 



Towards Employment WA group 



Month relative to random assignment 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 


NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample sizes may vary because of missing values. 



The WorkAdvance Study 
Table 3.3 

Year 2 Impacts on Participation in Primary Occupational Skills Training Program, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 


Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Started primary training 

71.6 

37.1 

34 5 *** 

66.2 

35.9 

30.4 *** 

60.2 

25.2 

34 9 *** 

64.0 

27.6 

36.4 *** 

Completed primary training 

60.5 

24.5 

35 9 *** 

55.7 

30.6 

25 i *** 

41.0 

16.1 

24 9 *** 

45.6 

17.8 

27 9 *** 

Currently in primary training 

4.7 

7.8 

-3.2 

3.9 

4.5 

-0.5 

10.7 

3.9 

6.8 *** 

10.3 

6.7 

3.6 

Dropped out of primary training 

6.4 

4.8 

1.6 

6.3 

0.7 

5.6 *** 

8.5 

5.2 

3.3 

7.8 

2.8 

5.0 *** 

Reasons for not completing primary 

training 












Found a job 

52.4 

19.8 

32.5 

33.3 

0.0 

33.3 

14.8 

2.3 

12.5 

20.1 

0.0 

22.8 

Not enough money to continue 

6.5 

6.5 

0.0 

33.3 

0.0 

33.3 

0.0 

40.1 

-44.9 

6.7 

20.0 

-13.3 

Needed to work 

22.5 

0.0 

22.8 

33.3 

0.0 

33.3 

14.7 

33.2 

-18.5 

0.0 

15.2 

-16.3 

Other 

15.1 

86.8 

-71.7 

41.7 

100.0 

-58.3 

75.6 

39.3 

36.3 

76.5 

74.3 

2.2 


Characteristics of primary training classes a 
Included examples/assignments 


relevant to career interests 

77.6 

59.3 

18.3 

67.1 

72.4 

-5.3 

75.7 

62.4 

13.2 

71.3 

63.1 

8.1 

Organized and clear expectations 

81.3 

74.1 

7.2 

83.6 

81.7 

1.9 

75.4 

75.5 

-0.1 

73.6 

78.6 

-5.0 

Involved hands-on experience 

83.9 

63.4 

20.5 

77.3 

72.9 

4.5 

81.3 

77.5 

3.7 

72.5 

72.1 

0.4 

Used skills learned in primary training 
on the job 

59.9 

20.0 

39 9 *** 

44.3 

23.0 

2 i 3 *** 

45.3 

14.8 

30.6 *** 

39.3 

14.1 

25.2 *** 

Participated in primary training while 
employed 

16.5 

15.4 

1.2 

12.4 

14.4 

- 2.1 

32.3 

16.2 

16.1 *** 

22.9 

11.6 

j 2 3 *** 

Employed and received pay for time in 
primary training * 3 

4.5 

5.3 

- 0.8 

3.6 

3.7 

- 0.2 

11.4 

7.0 

4.5 * 

5.9 

4.5 

1.4 

Paid for primary training out of 
pocket or with loans 

4.5 

17.8 

-13 4 *** 

8.3 

16.2 

_7 9 ** 

3.6 

12.7 

_9 j *** 

6.2 

13.7 

-7 5 *** 
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Table 3.3 (continued) 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Average amount spent on or borrowed 

for primary training ($) 

110 

623 

-512 *** 

227 

411 

-184 

279 

562 

-283 

193 

813 

-619 *** 

Amount spent or borrowed for 
primary training (%) 

None, never in primary training 

28.4 

62.9 

ttt 
-34 5 *** 

33.8 

64.5 

ttt 

-30 7 *** 

40.1 

75.2 

ttt 

-35 1 *** 

36.2 

73.0 

ttt 

-36.8 *** 

None, did not pay for primary training 

67.2 

19.3 

47 9 *** 

57.9 

19.3 

38.6 *** 

56.4 

12.7 

43.8 *** 

57.7 

13.4 

44.3 *** 

Under $5,000 

3.9 

13.5 

_g 5 *** 

6.9 

14.0 

_7 i ** 

1.8 

8.3 

-6.5 *** 

4.7 

8.3 

-3.6 * 

$5,000 or more 

0.5 

4.3 

-3 8 *** 

1.4 

2.3 

- 0.8 

1.7 

3.9 

- 2.2 

1.5 

5.4 

_4 0 ** 

Primary training financed by loans 

0.6 

6.2 

-5.6 *** 

1.5 

3.4 

-1.9 

1.8 

5.3 

-3.5 ** 

4.0 

6.5 

-2.5 

Received a reimbursement from an 

employer for primary training cost 

1.0 

0.4 

0.6 

1.0 

0.0 

1.1 

0.0 

0.9 

-0.9 * 

0.0 

0.3 

-0.3 

Received help paying for primary training 

from an outside organization^ 

39.3 

13.4 

25 9 *** 

41.3 

9.2 

52 i *** 

46.4 

7.8 

38.6 *** 

47.1 

8.6 

38.5 *** 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Primary occupational skills training for program group respondents is, for the most part, the skills training offered through WorkAdvance; for control group 
respondents, it is the first skills training after random assignment. All measures in this table relate to the primary skills training only. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 
categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

Italics indicate the metric is not among the full sample shown in the table. Therefore, the measure is nonexperimental and statistical tests were not 
performed. 

“Percentage represents respondents who indicated they "strongly agree" with the statements. 

b Measure includes participants who were paid by an employer; an employment, government, or community-based program; or in some other way. 

“Outside organizations include the WA provider and employment, government, and community-based organizations. 



WorkAdvance yielded double-digit, statistically significant impacts at all sites on par- 
ticipation in any education and training, ranging from 2 1 percentage points at St. Nicks Alliance 
to 27 percentage points at Madison Strategies Group. This overall measure includes the follow- 
ing types of educational activities: Adult Basic Education (ABE), General Educational Devel- 
opment (GED) classes, occupational skills training, college courses, and short-term classes. 

Overall, these impacts are noteworthy, given that a substantial proportion of control 
group members (between 50 percent and 63 percent) participated in education or training activi- 
ties. For example, at Towards Employment, WorkAdvance increased participation in education 
and training by approximately 26 percentage points above the control group level of 55 percent. 
As expected, these effects were mostly concentrated in occupational skills training, particularly 
training in the sectors targeted by WorkAdvance. 4 As shown in Box 3.2, community colleges 
and for-profit training providers were the most common places where control group members 
received training; the sector in which control group training occurred most commonly was 
health care. 

In general, WorkAdvance increased participation in occupational skills training by 30 
percentage points or more, depending on the site. The skills training completion and credential 
attainment rates for WorkAdvance group members were also significantly higher than those of 
control group members at each of the sites. Impacts in the likelihood of attaining occupational 
skills training credentials range from 23 percentage points for Madison Strategies Group to 39 
percentage points at Per Scholas, which offered its program enrollees an opportunity to acquire 
multiple information technology sector credentials. 

WorkAdvance also increased individuals’ participation in short-term classes at three 
sites (such as cardiopulmonary resuscitation [CPR], how to operate a forklift, certification in a 
computer software like Microsoft Word, or basic computer skills) that may have overlapped 
somewhat with the occupational skills classes. It is important to note that WorkAdvance did not 
appear to divert people who would have gone to college into vocational training instead. This 
would be evident if there were a large negative impact on participation in college courses 

Because WorkAdvance group members were more likely to have participated in an oc- 
cupational skills training program, it is not surprising that they were also significantly more like- 
ly to have spent more months, on average, in such programs. Among just those WorkAdvance 


4 Rates of participation in targeted sector training among WorkAdvance group members, as measured via 
the Year 2 Survey (Table 3.2), are lower than those measured via the providers’ MIS records (Table 2.5). Much 
of the discrepancy is probably due to recall error in the survey — a phenomenon that would have been present 
for both WorkAdvance and control group members. Appendix Table A.3 shows a sensitivity analysis that 
measures the extent to which training participation rates varied across data sources. Results show that the pat- 
terns found in providers’ MIS training records are largely consistent with those in WorkAdvance group mem- 
bers’ responses to the Year 2 Survey question about their participation in targeted sector training. 
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and control group members who attended occupational skills training, however, differences in 
the length of time spent in training were more modest. Among individuals who attended such 
training, control group respondents at the Per Scholas and St. Nicks Alliance sites spent about 
the same amount of time in training as WorkAdvance group respondents. WorkAdvance group 
training participants at Madison Strategies Group and Towards Employment spent one to two 
more months, on average, in occupational skills training than control group members who par- 
ticipated in training. 

Despite the fact that the length of time spent in training was roughly comparable for 
WorkAdvance and control group respondent training participants, there were some differences 
in when individuals engaged in training. Specifically, control group respondents generally did 
not enroll in an occupational skills training program immediately after random assignment. This 
is illustrated in Figure 3.2, which shows that the percentage of WorkAdvance group respond- 
ents participating in occupational skills training was higher than that of control group respond- 
ents in the first six months after random assignment. 

Participation impacts on occupational skills training lasted later into the follow-up peri- 
od for Madison Strategies Group and Towards Employment, the two providers that operated 
dual-track programs until late 2012. Both Madison Strategies Group and Towards Employ- 
ment’s WorkAdvance group respondents were more likely than their control group counterparts 
to be enrolled in a training program through the first 18 months after random assignment (Fig- 
ure 3.2), suggesting that these providers engaged at least some program enrollees in training 
relatively late in the follow-up period. 5 

More detailed survey questions were asked of sample members about their primary oc- 
cupational skills training; these findings are presented in Table 3.3. 6 Given the very large and 
statistically significant differences between the WorkAdvance group and the control group in 
participation in occupational skills training, the significant effects of WorkAdvance on primary 
training participation and completion are not surprising. These more detailed questions shed 
light on how the WorkAdvance training might have been different (or not) from training that 
control group members accessed. For example, among just those individuals who did participate 
in training, WorkAdvance group members, compared with control group members, were more 


5 A supplementary analysis of training participation in Months 10 through 18 after random assignment 
found that some early program enrollees at Madison Strategies Group and Towards Employment were attend- 
ing a WorkAdvance training program at that stage. This finding suggests that these providers reengaged at least 
some early program enrollees after making the transition to a training-first approach. It is also possible that 
WorkAdvance group members were engaged in a lengthy occupational skills training program (for example, 
diesel mechanics training offered at the Madison Strategies Group site, which spanned eight months). 

6 This refers to the occupational skills training offered through the WorkAdvance provider for WorkAd- 
vance group members and the first occupational skills training program after random assignment for control 
group members. 
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likely to “strongly agree” that their training included examples or assignments relevant to their 
career interests (at all sites but St. Nicks Alliance) and, at Per Scholas, involved hands-on expe- 
rience. 


In addition, at each of the sites, WorkAdvance group respondents were much more like- 
ly than control group respondents to have used skills learned in their primary training on a job. 
(See Table 3.3.) Furthermore, at Madison Strategies Group and Towards Employment, Work- 
Advance group respondents were more likely to have participated in the primary training while 
employed. This finding may be partially explained by the fact that WorkAdvance group re- 
spondents at these two sites were also more likely than their respective control group respond- 
ents to be in training at the time of the survey interview (when many individuals were em- 
ployed). Again, both Madison Strategies Group and Towards Employment made a transition in 
the last couple of years of WorkAdvance implementation to a training-first approach, which 
may have resulted in some program enrollees participating in training programs later into their 
respective service periods. 

• WorkAdvance (which offered training and other services free of charge 
to enrollees) reduced sample members’ expenditures for training. 

Publicly funded resources for training — made available for job seekers through the 
Workforce Investment Act (WIA) — were significantly reduced during the study period, having 
declined nearly 60 percent from 2000 to 20 10. 7 Perhaps for this reason, control group members, 
relative to WorkAdvance group members, were much more likely to have spent or borrowed 
money for training — of any type — at all sites. In addition, WorkAdvance significantly re- 
duced the amount survey respondents spent or borrowed for training. WorkAdvance reduced 
the occurrence of education-related spending or borrowing by a range of 8 percentage points at 
Towards Employment and St. Nicks Alliance to 13 percentage points at Per Scholas. These ef- 
fects are statistically significant across all the sites (Table 3.3). Per Scholas and Madison Strate- 
gies Group control group respondents, in particular, were more likely to have financed their 
primary training with loans. Among control group respondents who paid for their primary train- 
ing, the average amount spent or borrowed ranged from $2,300 at St. Nicks Alliance to $5,882 
at Towards Employment (not shown). 8 


’Hilliard (2013). 

s Appendix Table D.l shows impacts on a broader array of education and training measures. 
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Impacts on Job Search Activities, by Site 

• Relative to control group members, WorkAdvance group members at 
each site were not more likely to have looked for a job, but they were 
much more likely to report receiving help with a job search at some 
point during the follow-up period (Table 3.4). 

Table 3.4 presents impacts on job search and work-based learning activities, as meas- 
ured using responses to the Year 2 Survey. WorkAdvance did not significantly affect the pro- 
portion of people who looked for a job during the follow-up period. 9 WorkAdvance did, how- 
ever, produce statistically significant effects on participation in work-based learning: 10 Work- 
Advance group members at three of four sites were much more likely than their control group 
counterparts to participate in on-the-job training and were also more likely to participate in an 
internship. In the wake of the Great Recession, employers were very selective about hiring indi- 
viduals with relevant experience. WorkAdvance providers reacted to this by developing intern- 
ships to build up participants’ resumes with sector-relevant experience. 

The survey also asked about the receipt of help related to a job search, in three forms: 
finding a job in the targeted sector, finding a specific job opening, and scheduling a job inter- 
view. While WorkAdvance group members were much more likely than control group mem- 
bers to say that they received help in each of these areas, increases were largest for receiving 
help finding a job in the targeted sector, with impacts ranging from 42 percentage points at To- 
wards Employment — which relied on intermediary organizations that had existing relation- 
ships with sector employers to match program enrollees with job leads — to about 56 percent- 
age points at Per Scholas, which used its strong relationships with large finns to open doors that 
training participants might not have had access to in the absence of the program. This finding 
supports the WorkAdvance providers’ common goal of getting WorkAdvance group members 
into jobs in the sectors they were targeting, and not just into “any job.” 

Generally, between 20 percent and 42 percent of WorkAdvance group members report- 
ed that they found at least one job with help from the WorkAdvance program. In contrast, con- 
trol group members reported that they found at least one job with help from another source, es- 
pecially friends or relatives. 


9 While Table 3.4 shows that WorkAdvance members at the Madison Strategies Group site are less likely 
than their respective control group members to have ever looked for a job, this 7 percentage point impact repre- 
sents a difference of only about six respondents. 

10 Work-based learning activities include on-the-job training and internships. 
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The WorkAdvance Study 
Table 3.4 


Year 2 Impacts on Participation in Job Search Activities, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Ever looked for a job a 

83.6 

86.7 

-3.1 

84.8 

81.5 

3.3 

75.2 

82.7 

-7.6 ** 

82.4 

82.4 

0.0 

Individual job search 

77.3 

79.4 

- 2.1 

73.5 

75.2 

- 1.6 

68.5 

73.4 

-5.0 

75.2 

76.6 

-1.4 

Not working and has looked for 

a job in the past four weeks * 1 

21.1 

28.1 

-7.1 * 

25.4 

27.3 

-1.9 

15.5 

17.2 

-1.7 

24.9 

26.5 

- 1.6 

Ever participated in an internship 

31.0 

15.1 


11.8 

7.2 

4.7 

11.0 

4.0 

7 0 *** 

23.6 

4.9 

lg 7 *** 

Ever participated in on-the-job training 

4.0 

7.8 

-3.8 * 

5.9 

3.3 

2.6 

6.1 

2.2 

39 ** 

6.8 

2.0 

4 g *** 

Job search activities in which respondent received 
Job search activities 81.7 

help 

36.2 

45.5 *** 

75.1 

44.8 

30.3 *** 

76.8 

24.9 

31 g *** 

72.5 

33.4 

39 i *** 

Finding a job in targeted sector 

70.9 

14.6 

56.3 *** 

52.7 

11.1 

41 6 *** 

61.0 

8.2 

52.8 *** 

59.7 

18.2 

41 5 *** 

Finding a specific job opening 

69.8 

29.2 

40.6 *** 

60.3 

34.2 

26.0 *** 

65.8 

19.4 

46 4 *** 

61.0 

26.0 

35.0 *** 

Scheduling a job interview 

64.4 

20.3 

44 2 *** 

55.8 

24.6 

31 2 *** 

53.5 

7.9 

45.5 *** 

51.4 

17.1 

34.3 *** 

Obtained a job with help from: 

WorkAdvance provider 

42.8 

0.5 

42 3 *** 

23.6 

0.0 

24.2 *** 

31.4 

0.7 

3Q 7 *** 

21.2 

1.2 

20.1 *** 

Employment agency or CBO c 

19.0 

20.7 

-1.7 

24.2 

27.7 

-3.5 

24.9 

34.6 

_9 g ** 

17.6 

21.2 

-3.6 

Own job search 

38.8 

45.1 

-6.4 

45.5 

47.5 

- 2.0 

50.5 

55.8 

-5.4 

51.4 

57.1 

-5.6 

Social media 

3.8 

3.0 

0.8 

5.9 

2.2 

3.7 * 

6.5 

5.6 

0.9 

1.4 

3.3 

-1.9 

Other 1 * 

34.1 

51.1 


39.2 

43.2 

-3.9 

38.8 

42.4 

-3.7 

36.7 

40.7 

-4.0 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



(continued) 



Table 3.4 (continued) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; CBO = community-based 
organization. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Sample sizes may vary because of missing values. 

a "Looked for a job" may include group or individual job search. 

b Job search activities within the last four weeks include contacting employers, contacting a 
public employment agency, contacting a private employment agency, contacting a school or 
university employment center, contacting friends or relatives, attending or enrolling in a program 
or course, attending job training, interviewing for a job, sending out a resume or filling out an 
application, checking union or professional registers, looking at job advertisements, placing an ad 
or answering an ad, and other job search activities. 

“Category includes a One-Stop Career Center, a staffing or temporary employment agency, or 
some other employment agency or program. 

d "Other" includes friends, relatives, and the response "Other." 


Impacts on Postemployment Help Received, by Site 

• WorkAdvance group members at each site were much more likely than 
control group members to report receiving postemployment services, 
with the impacts ranging from approximately 40 percentage points at 
Per Scholas to more than 70 percentage points at Madison Strategies 
Group (Table 3.5). 

Table 3.5 presents effects on postemployment service help received from staff members 
at employment, government, or community-based organizations. It is somewhat surprising that 
WorkAdvance significantly increased the receipt of postemployment help, given the relatively 
late start (discussed in Chapter 2) that providers had in putting this program component into 
place. Although there are statistically significant increases across all the sites, Madison Strate- 
gies Group’s increase is impressively large. The result for this site may be explained in part by 
the provider’s perfonnance-based financial compensation for WorkAdvance staff members who 
were responsible for delivering postemployment services, also described in Chapter 2. 

The Year 2 Survey asked a series of questions about the nature of postemployment ser- 
vices received, including help that staff provided through direct contact with the respondent, as 
well as help provided through direct staff contact with the respondent’s employer, to the extent 
that the respondent was aware of such contact. Among WorkAdvance group respondents, a lit- 
tle less than 50 percent at Per Scholas and more than 60 percent at St. Nicks Alliance, Madison 
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The WorkAdvance Study 
Table 3.5 

Year 2 Impacts on Participation in Postemployment Services, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 





Difference 



Difference 



Difference 



Difference 

Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Received a postemployment service 3 

64.3 

25.5 

3g g *** 

69.2 

23.6 

45.6 *** 

85.8 

14.3 

71 6 *** 

68.8 

23.3 

45.6 *** 

Postemployment service areas in which respondent received help 










Keeping current or most recent job 

31.3 

5.0 

26.3 *** 

20.3 

3.0 

17 4 *** 

31.3 

4.5 

26.8 *** 

27.1 

3.0 

24 i *** 

Discussing how job is going 

Discussing how to handle problems 

46.9 

12.9 

34.0 *** 

62.1 

11.5 

50.6 *** 

75.2 

7.9 

67.3 *** 

61.1 

13.3 

47 g *** 

with coworkers or supervisors 

36.3 

9.4 

26.8 *** 

38.5 

8.4 

30.1 *** 

49.2 

2.8 

45 4 *** 

44.3 

8.9 

35 4 *** 

Respondent aware that employer was contacted to: 












Discuss problems with respondent's 
performance on the job 

Collect feedback on respondent's 

10.6 

4.8 

5.8 ** 

13.1 

4.8 

g 4 *** 

22.9 

1.5 

2 i 3 *** 

14.9 

4.5 

10.5 *** 

performance/promotion possibilities 
Learn more about opportunities for 

10.8 

8.3 

2.6 

15.7 

5.7 

10 i *** 

32.1 

2.2 

29 9 *** 

17.9 

6.0 

119 *** 

advancement 

12.7 

7.8 

4.9 * 

17.3 

7.9 

g 4 *** 

27.0 

3.4 

23 7 *** 

14.7 

5.7 

9 o *** 

Help respondent keep job 

8.4 

5.5 

2.8 

13.7 

4.8 

g 0 *** 

17.8 

2.5 

15 3 *** 

10.9 

5.1 

5.8 ** 

Number of times respondent was aware that employer was contacted 










to discuss respondent 






ttt 



ttt 



ttt 

Never 

80.8 

87.0 

-6.2 

73.6 

88.2 

-14.6 *** 

58.4 

94.4 

-36.0 *** 

73.3 

90.4 

_17 i *** 

Once or twice 

15.1 

11.0 

4.1 

21.1 

8.8 

12 4 *** 

32.8 

4.7 

28.2 *** 

19.6 

8.1 

11.5 *** 

More than 2 times 

4.1 

2.0 

2.1 

5.3 

3.0 

2.2 

8.8 

0.9 

7 g *** 

7.1 

1.5 

5.6 *** 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 
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Table 3.5 (continued) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Sample sizes may vary because of missing values. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
10 percent. The Westfall- Young adjusted p-values were used for categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across 
research groups. Statistical significance levels are indicated as follows: f f f = 1 percent; f f = 5 
percent; f = 10 percent. 

a Postemployment services include those received while an individual is employed: 
Respondent obtained help keeping current or most recent job, or employment service providers 
contacted the employer about how the respondent could keep the job or advance. 


Strategies Group, and Towards Employment reported that they ever received help via check-ins 
about “how their job was going.” Though relatively few respondents reported that a staff mem- 
ber from the program had ever contacted their employer to discuss their employment, WorkAd- 
vance group respondents were still more likely than control group respondents to indicate that 
such contact occurred. At St. Nicks Alliance, Madison Strategies Group, and Towards Em- 
ployment, WorkAdvance group respondents were much more likely than control group re- 
spondents to report that a staff person contacted their employers specifically to discuss job per- 
formance problems, as well as to discuss how to help the respondent keep his or her job. As dis- 
cussed in Chapter 2, Per Scholas had reservations about reaching out to employers in this way 
as it might signal that the participant was potentially a “problem employee.” 

The number of times staff members contacted employers also tended to differ between 
WorkAdvance and control group members. For example, at both the Madison Strategies Group 
and Towards Employment sites, WorkAdvance group respondents were more likely than their 
respective control group respondents to report more than two occasions when their employer 
was contacted. 


Participation Impacts by Enrollment Cohort Varied 

As mentioned in Chapter 1 , a number of changes were implemented during and after the fall of 
2012 across all sites in an effort to improve service delivery, most notably a greater focus on 
advancement coaching (getting the postemployment service component into place) and, for two 
of the providers (Madison Strategies Group and Towards Employment), moving to a training- 
first approach, which meant that most, if not all, new WorkAdvance group members were rout- 
ed toward occupational skills training before beginning a job search. Given these changes in the 
scope and, perhaps, quality of services that the WorkAdvance providers implemented, a natural 
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question is whether there was any subsequent change in the pattern of participation impacts for 
those who enrolled earlier (the “early cohort’ ’) compared with those who enrolled later (the “late 
cohort”). (As will be seen in the analyses in the following chapters, the economic impacts are 
different for the two cohorts in some sites.) Table 3.6 helps answer this question by showing 
two-year participation impacts for individuals who enrolled in the study between June 201 1 and 
September 2012 and for those who enrolled between October 2012 and June 2013. 

• At least some of the WorkAdvance participation impacts are larger for 
the late cohort, compared with the early cohort, but these differences — 
in terms of their magnitude — vary by site (Table 3.6). 

For Per Scholas, WorkAdvance’s impacts on employment service receipt are about the 
same for the early and late cohorts, including impacts on ever receiving postemployment ser- 
vices. There is a somewhat larger impact, however, for the late cohort on the likelihood of start- 
ing training in the targeted sector: The impact was 43 percentage points for the early cohort and 
56 percentage points for the late cohort. 11 

For the St. Nicks Alliance site, there are larger impacts for the late cohort on the receipt 
of help with a job search: The difference between the WorkAdvance group and the control 
group in receiving job search help is 20 percentage points for the early cohort and 37 percentage 
points for the late cohort. Participation impacts on other activities are very similar for the early 
and late cohorts. 

While Madison Strategies Group’s impacts on training participation were larger for the 
late cohort than for the early cohort, it was surprising that the difference in the two cohorts’ 
training impacts was not statistically significant, given the provider’s transition over time to the 
training-first approach for all enrollees. One partial explanation might be that some WorkAd- 
vance group members in the early cohort participated in training later in their respective service 
periods (Figure 3.2). In contrast to Madison Strategies Group, late cohort impacts at Towards 
Employment — another provider that was initially operating a two-track program — are much 
larger than early cohort impacts regarding participation in employment services, especially 
those related to career planning and job readiness (not shown), finding a job in a particular field, 
and postemployment services. 


n For this cohort analysis, unless noted otherwise, impacts across cohorts are discussed only if they are sta- 
tistically different from one another. 



The WorkAdvance Study 
Table 3.6 

Selected Participation Characteristics of Survey Respondents, by Site and Cohort 




Early Cohort 


Late Cohort 




Difference 



Difference 

Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) Sig. 

Per Scholas 

Career readiness service 3 

88.6 

57.1 

31.6 *** 

88.8 

64.5 

24 3 *** 

Job search service 13 

80.9 

32.6 

48.3 *** 

81.2 

41.2 

40 j *** 

Postemployment service 3 

62.0 

23.2 

33 9 *** 

65.5 

29.2 

36.3 *** 

Started a skills training program 
in targeted sector 

Completed a skills training program 

67.9 

24.8 

43 i *** 

71.6 

15.3 

56.2 *** f 

in targeted sector 

Obtained a credential from a training program 

57.5 

17.2 

40.3 *** 

58.5 

6.8 

51.6 *** 

in targeted sector 

53.6 

11.2 

42.4 *** 

54.6 

4.7 

49 9 *** 

Sample size (total = 552) 

152 

132 


135 

133 


St. Nicks Alliance 

Career readiness service 3 

89.1 

63.1 

26.0 *** 

88.8 

68.2 

20.6 *** 

Job search service 13 

72.7 

52.5 

20.2 *** 

76.0 

39.2 

36.7 *** 

Postemployment service 3 

70.7 

25.1 

45.5 *** 

64.5 

25.5 

39 o *** 

Started a skills training program 
in targeted sector 

Completed a skills training program 

57.4 

20.7 

36.7 *** 

61.7 

23.3 

38.4 *** 

in targeted sector 

Obtained a credential from a training program 

51.8 

17.9 

23 9 *** 

50.6 

23.8 

26.9 *** 

in targeted sector 

49.3 

13.8 

35.5 *** 

46.2 

22.2 

24.0 *** 

Sample size (total = 384) 

106 

88 


99 

91 



(continued) 


Conclusion 

As demonstrated throughout this chapter, across all sites WorkAdvance substantially increased 
the take-up of employment services and occupational skills training participation, completion, 
and credential attainment. The effects of WorkAdvance on skills training participation are, as 
expected, most evident for training related to the targeted sector. Below are some key site- 
specific findings about participation impacts to keep in mind when reviewing the economic im- 
pacts in Chapter 5. 
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Table 3.6 (continued) 




Early Cohort 


Late Cohort 





Difference 



Difference 


Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Madison Strategies Group 

Career readiness service 3 

89.6 

40.6 

43 9 *** 

90.7 

46.1 

44 6 *** 


Job search service 15 

74.1 

24.3 

49 8 *** 

79.1 

25.5 

53.6 *** 


Postemployment service 15 

82.5 

12.3 

70 3 *** 

88.7 

15.7 

73 0 *** 


Started a skills training program 
in targeted sector 

Completed a skills training program 

50.9 

15.4 

35.6 *** 

59.2 

11.5 

47 7 *** 


in targeted sector 

Obtained a credential from a training program 

37.8 

9.5 

28 3 *** 

41.1 

6.6 

34 5 *** 


in targeted sector 

28.4 

7.7 

20.6 *** 

35.7 

5.9 

29 9 *** 


Sample size (total = 560) 

138 

115 


159 

148 



Towards Employment 

Career readiness service 3 

87.5 

61.9 

25.5 *** 

87.2 

41.6 

45.6 *** 

fff 

Job search service 15 

71.5 

44.4 

27 j *** 

73.9 

21.3 

52.5 *** 

fff 

Postemployment service 15 

64.8 

29.2 

35 7 *** 

72.6 

17.4 

55.2 *** 

ff 

Started a skills training program 
in targeted sector 

Completed a skills training program 

58.5 

16.7 

41.8 *** 

63.2 

14.5 

43 7 *** 


in targeted sector 

Obtained a credential from a training program 

44.8 

12.8 

32 \ *** 

49.7 

10.0 

39 7 *** 


in targeted sector 

32.7 

12.4 

20 3 *** 

39.3 

7.9 

4 *** 


Sample size (total = 562) 

144 

142 


142 

134 




SOURCE: MDRC calculations from the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 
includes all sample members randomly assigned in or after Quarter 4, 2012. 

Sample sizes may vary because of missing values. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated 
as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

““Services include career planning, job readiness, and services related to keeping or advancing on a job. 
b Services include help finding a job in the target field, finding a specific job opportunity to apply for, and 
scheduling a job interview. 

c Postemployment services include those received while an individual is employed: Respondent obtained help 
keeping current or most recent job, or employment service providers contacted the employer about how the 
respondent could keep the job or advance. 
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Per Scholas, which focused on the information technology (IT) sector in New York 
City, yielded the largest impacts on occupational skills training starting, completion, and cre- 
dential attainment. Also, relative to the other sites, Per Scholas produced the largest increases in 
people reporting that they received help in finding a job in the targeted sector, which may reflect 
Per Scholas’s many years of experience preparing individuals for careers in the IT sector in 
New York City. 

St, Nicks Alliance, which focused on the environmental remediation and pest control 
sectors in New York City, increased participation in every major category of employment and 
training services (career advice, starting occupational skills training, job search, and postem- 
ployment), but most of its impacts in these categories were the smallest of the four WorkAd- 
vance sites. This is, at least in part, due to relatively high control group service participation 
rates at this site, especially with respect to skills training in the targeted sector. This suggests 
that the WorkAdvance program at St. Nicks Alliance had a higher bar to meet in delivering pro- 
gram services with a value beyond what enrollees could obtain elsewhere, relative to the other 
three sites. 

Madison Strategies Group, which focused on the transportation and (later) the manu- 
facturing sectors in Tulsa, produced the largest impact in every category of employment ser- 
vices. At least two factors may have contributed to these large effect sizes: (1) Relative to the 
other three sites, control group receipt rates of these services were lower at this site, and (2) 
Madison Strategies Group employed a performance-based financial compensation model for its 
staff. WorkAdvance group members at this site were also more likely than their respective con- 
trol group members to be enrolled in training later in the follow-up period — 10 to 18 months 
after random assignment. This may reflect the provider’s transition to a “training-first for all” 
approach, which may have induced some WorkAdvance group members in the early cohort to 
begin training late in their service periods. 

Towards Employment, which focused on health care and manufacturing in the Greater 
Cleveland area, was the other provider that transitioned to a “training-first for all” approach in 
fall 2012. At Towards Employment, WorkAdvance group members were also more likely than 
their respective control group members to be enrolled in training 10 to 18 months after random 
assignment. The participation impacts on several types of employment services at Towards 
Employment were much larger for its late cohort than for its early cohort. 

Many employment and training programs that have been studied have produced only 
small or no impacts on participation in services. It is not always easy for program enrollees to 
attend program activities, and it is certainly often difficult for them to finish long activities like 
training. The bottom line in this chapter is a very positive one: Across all sites, compared with 
what would have happened in the absence of the program, WorkAdvance resulted in very large 
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and statistically significant increases in participation in every category of measured Work- 
Advance-type services. This provides the field with an excellent opportunity to understand the 
effectiveness of these services. But how much did it cost to provide all these services to the typ- 
ical WorkAdvance participant, and how does this compare with the costs of the services that 
individuals would have received without WorkAdvance? This topic, which is another indication 
of the difference in treatment provided by WorkAdvance, is discussed in the next chapter. 
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Chapter 4 

The Cost of the WorkAdvance Program 


Chapter Highlights 

• Expenditures on the operation of WorkAdvance ran between $5,200 and 
$6,700 per participant for the four providers delivering the program and were 
between $5,200 and $5,800 at three of the programs. 

• The largest share of gross program operating expenditures (roughly half) for 
each of the four providers went toward providing preemployment activities 
and occupational skills training. 

• Net costs — computed by subtracting the costs that would have occurred in 
the absence of WorkAdvance from the costs of operating the program — are 
smaller than operating costs, but only the net costs of Per Scholas appear to 
be significantly smaller. (More Per Scholas control group members enrolled 
in training and they generally spent more hours in training than did control 
group members at the other sites.) Net costs are about $3,500 per participant 
for Per Scholas, and in the range of $4,900 to $5,900 at the other three sites. 

• Differences between gross in-program operating costs and net costs are al- 
most entirely attributable to control group members enrolling in community 
colleges and for-profit providers, and in the case of Per Scholas, in four-year 
colleges as well. 


Introduction 

The previous chapter presents infonnation on the receipt of services by WorkAdvance group 
members, which it compares with the receipt of other available services by control group mem- 
bers. This chapter examines a different aspect of the contrast between the two research groups: 
what it cost to provide WorkAdvance’s services, and how this compares with the cost of what 
would have happened in the absence of WorkAdvance. This cost analysis does not take into 
account the economic benefits produced by WorkAdvance (discussed in the following chapter). 
A full benefit-cost analysis, using follow-up data from a much longer period than that available 
for this report, is planned for the future. 

The following section of this chapter defines the key cost terms used in the analysis and dis- 
cusses a few issues that arose in the attempt to estimate the costs of WorkAdvance. The next 
section describes the sources of the data and the methods used to measure costs. Findings from 
the cost analysis are then presented, followed by a brief conclusion. 
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Key Concepts 


Gross Costs Versus Net Costs 

It is important to distinguish between two measures of costs: gross costs and net costs. 
Because these measures serve different purposes, estimates of both are reported in this chapter. 

Gross costs are simply the outlays required to operate a program. An estimate of gross 
costs is needed to plan a budget and is obviously essential for providers that are already operat- 
ing WorkAdvance and want to continue the program. A city without WorkAdvance that is con- 
sidering introducing the program should find estimates of the cost of operating similar programs 
elsewhere to be of value. In addition, gross cost estimates can be used to examine how program 
resources are allocated among the program’s critical components, such as engaging with em- 
ployers, recruiting program participants, screening potential participants, and then providing 
soft skills training, occupational skills training, interview training, and postemployment reten- 
tion and advancement activities. 

Net costs are the changes in costs that result from operating a program. Stated slightly 
differently, they are costs that would not be incurred in the absence of the program. The net 
costs of WorkAdvance differ from the program’s gross costs because if the program did not 
exist, some program participants would have received at least a portion of the services provided 
by the program from other providers. For example, individuals may have found soft skills train- 
ing or occupational skills training on their own elsewhere, thereby engendering costs. Net costs 
are estimated as the difference between the cost of services received by a typical WorkAdvance 
participant and the cost of similar services received by an average control group member. It fol- 
lows that net costs for WorkAdvance (but not gross costs) can appropriately be compared with 
the program’s net economic impacts, because both measure changes that would not have oc- 
curred without the program. 

Steady-State Costs 

One objective of the cost analysis is to determine what the future cost of WorkAdvance 
would be if it were to become an ongoing program — specifically, what the program would 
cost once it reached a “steady state” of operation. Because WorkAdvance was implemented as a 
demonstration program that is being evaluated by random assignment, certain research costs 
were incurred that are excluded from the cost estimates presented here because they would not 
nonnally exist in an ongoing version of the program. Such excluded costs encompass payments 
to the evaluation firm (MDRC) for such things as technical assistance, costs incurred that relate 
to performing random assignment (such as time spent explaining the details of research and 
random assignment to those randomly assigned), and efforts associated with recruiting and 
screening the control group. 
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The data used in the cost analysis were taken from a three-year period and divided into 
three sequential 12-month periods corresponding to the first three years of the program’s opera- 
tion: April 1, 2011, through March 31, 2012 (Year 1); April 1, 2012, through March 31, 2013 
(Year 2); and April 1, 2013, through March 31, 2014 (Year 3). This report emphasizes findings 
for Year 2, although some findings for Years 1 and 3 are also reported. 

Year 2 of the program was selected as the steady-state period for the WorkAdvance 
cost analysis. By Year 2, many of the program’s start-up costs, related to its staff members 
learning how to operate most effectively and efficiently and establishing relationships with em- 
ployers in the targeted occupational sectors, had substantially diminished. At the same time, 
during Year 2 many program participants graduated from occupational skills training and 
reached the postemployment stage of WorkAdvance. Thus, costs during the year reflect both 
the pre- and postemployment stages of program operations. It should be noted that Year 2 does 
not represent an ideal steady state, but it is the best available option to serve as one. 1 

Aggregate Costs and Costs per Program Participant 

Some of the estimates presented in this chapter are reported as costs per program partic- 
ipant. More specifically, aggregate costs for Year 2 are divided by the number of program en- 
trants in that year. As demonstrated in Appendix E, this provides an estimate of costs per partic- 
ipant for those entering WorkAdvance during Year 2 over their entire 24 months of program 
eligibility. Essentially, this occurs because a consistent flow of individuals entering and exiting 
the program balances out in a steady state. While entrants during Year 2 incurred only 12 or 
fewer months of costs during that year, some program participants who entered during Year 1 
continued to incur costs during Year 2. As detailed in the appendix, the costs that these earlier 
entrants incurred in Year 2 represent (and can substitute for) the costs that Year 2 entrants in- 
curred after their year of entry ended. 

Of course, some WorkAdvance entrants incurred more costs than others because they 
took greater advantage of the services offered by the program, and some individuals who en- 
tered the program dropped out of the program before incurring many WorkAdvance costs. 
Thus, the estimates should be viewed as the cost of serving an average or typical program par- 
ticipant. 


The division of aggregate net costs by the number of individuals potentially eligible to 
incur them (WorkAdvance group members) allows for an appropriate comparison with Work- 
Advance’s economic impacts, which are estimated as averages for all individuals who were 
randomly assigned to the WorkAdvance group and thus potentially eligible to receive program 
services (regardless of the extent to which they actually took advantage of them). 


'Potential disadvantages to the selection of Year 2 as a steady state are described in detail in Appendix A. 
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The division of aggregate gross costs by the number of individuals potentially eligible 
to incur them (WorkAdvance group members) enables comparisons of costs among the four 
WorkAdvance sites, which differ in scale. The gross cost measure also facilitates planning for 
WorkAdvance programs in places other than the four evaluation sites because it indicates the 
costs of operating the program for each individual who is expected to participate. A rough pre- 
diction of aggregate costs for a new provider at a given scale can be obtained by simply multi- 
plying the measure by the anticipated number of participants. 

In-Program Versus Out-of-Program Costs 

Another way of breaking down costs that programs can engender is to sort them into in- 
program costs and out-of-program costs. In-program costs are those required to operate a pro- 
gram and are paid for directly out of program funds. Out-of-program costs are paid for through 
other sources. For example, out-of-program costs can arise if a program sends participants to 
outside organizations and those organizations absorb any resulting costs, or if the program uses 
volunteers or donated resources. Other than occasionally prevailing on employers to talk to par- 
ticipants about job opportunities and needed skills in their respective sectors, WorkAdvance 
engendered almost no costs of this sort. However, as seen in Chapter 3, some control group 
members (as well as some members of the WorkAdvance group) enrolled in vocational training 
on their own and received pre- and postemployment services from providers other than Work- 
Advance. The costs that these activities led to are taken into account in estimating net costs. 


Data Sources and Methods 

Calculating the costs of WorkAdvance required that cost data from the four sites (Per Scholas, 
St. Nicks Alliance, Madison Strategies Group, and Towards Employment) first be collected and 
sorted appropriately. Once pertinent cost data were obtained, they were then categorized, fil- 
tered, and distributed so that they could be analyzed. Aggregate costs were then broken down 
further into per participant measures for each site. 

Data Sources 

As discussed above, gross costs are a fairly straightforward tally of the resources ex- 
pended in the delivery of a program. In the case of WorkAdvance, virtually all program costs 
corresponded to the set of expenditures authorized in the providers’ official program budgets, 
which were established before each program year. Each provider was required to submit period- 
ic financial reports throughout the life of the program, enumerating all monetary expenditures 
incurred in line with its respective WorkAdvance budget. 2 Conversations with the WorkAd- 

2 Per Scholas, St. Nicks Alliance, and Towards Employment submitted reports quarterly, while Madison 
Strategies Group submitted reports monthly. 


96 



vance program director at each site also took place to assess whether there were any costs unac- 
counted for in the financial reports (such as unbudgeted expenses or services delivered by out- 
side providers). 3 

Net costs were more difficult to capture. They are estimated by removing from gross 
costs any costs that would have existed in the absence of WorkAdvance. These were estimated 
as the costs of preemployment services and support, occupational skills training, and postem- 
ployment retention and advancement services that were received by members of the control 
group. 4 Net cost estimates were then developed by subtracting the costs of services received by 
a typical control group member from the gross costs of a typical WorkAdvance participant. 

The number of WorkAdvance participants at the different sites was drawn from files 
provided by each site. The files were used to determine the number of participants at each site in 
Year 2. This number was used as a denominator in the computation of per participant costs. 

Cost Categories 

After gross costs were ascertained, they were categorized to facilitate further analysis. 
Gross costs were classified on two separate fronts — by primary program components and by 
key budget descriptors. WorkAdvance program activities were broken down into six core func- 
tions: recruitment; job development and employer engagement; screening; preemployment ser- 
vices and support; occupational skills training; and postemployment retention and advancement. 
Because certain costs may straddle one or more of those functions or may not be easily attribut- 
able to any single one (such as supervisory costs, office space, and so forth), additional catego- 
ries were added for management and for costs of an undefined nature. The categorization of 
recorded costs into these eight categories was done in consultation with the program directors at 
each site and MDRC implementation staff members. From a practical operational perspective, it 
is also useful to be able to view costs in terms of salary expenditures, fringe benefits, consultant 
services, overhead, and other expenses, so costs were also broken down in that manner. 5 


3 Per Scholas was the only provider to indicate that significant costs had not been incorporated into its fi- 
nancial reports. It identified two full-time staff members who had not been budgeted but who spent all their 
time on WorkAdvance, as well as additional overhead costs that had not been budgeted and that were therefore 
left out of its financial reports. 

4 Some members of the WorkAdvance group also participated in these activities outside the program, re- 
sulting in out-of-program costs. These costs are taken into account in estimating net costs. 

5 Certain costs could not easily be associated with any specific category. As an example, fringe benefits 
were reported by providers as a total amount incurred during a defined reporting period rather than as exact 
amounts corresponding to the various salary lines reported in that period. Therefore, fringe benefits were simp- 
ly prorated to the different cost categories on the basis of the percentage of total salary costs accounted for by 
each category. Similarly, all costs categorized as “undefined” were ultimately prorated to the seven remaining 
categories after all the identified costs had been distributed. Indirect costs, which are an institutional overhead 
rate built into a program budget, were similarly prorated across the same seven categories. 
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Findings 


Aggregate Gross Costs 

Table 4. 1 presents estimates of the aggregate in-program gross costs — gross costs that 
have not yet been divided by the number of program participants — that were incurred in oper- 
ating WorkAdvance at each of the program’s four sites during each of the three years for which 
data are available. Total dollar costs are reported in the top row of the panel for each site. These 
aggregate dollar values are indicated in percentage terms for seven categories that capture all the 
program activities in WorkAdvance. The seven categories are defined in Box 4. 1 . 

The aggregate dollar values are not usefully compared across the four sites because they 
reflect the differences among the providers both in costs per program participant and in the 
number of participants. But they can be compared across program years for each site, and it is 
appropriate to compare the percentage breakdown of how program resources were allocated 
among the seven categories across sites. 

• Aggregate costs tended to be larger in Year 2 than in either Year 1 or 
Year 3. 

Table 4. 1 indicates that aggregate gross costs in Year 2 were considerably larger than in 
Year 1 for all four sites, reflecting the fact that the programs did not begin operating until part of 
the way through Year 1 and had relatively few participants once they did start. Costs were also 
larger in Year 2 than in Year 3 at three of the four sites and were almost as large at the fourth 
site (Towards Employment). This is probably due to the break in enrollment of new program 
participants that followed the end of random assigmnent in June 2013, which lasted from four to 
eight months at the different sites. This is a major factor in the selection of Year 2 costs as being 
most representative of steady-state program operations. For that reason, the remainder of this 
chapter focuses mainly on that year. 

• Occupational skills training accounted for the largest share of operating 
costs. Preemployment activities, which like skills training promoted job 
readiness, were also significant. 

Table 4.1 reveals that with relatively few exceptions (discussed below, as the findings 
for each cost category are described), the four WorkAdvance providers distributed their re- 
sources similarly across the cost categories during Year 2. This can be seen clearly in Figure 
4.1, which uses a bar graph to show the percentage of aggregate costs that each provider spent 
on the respective program activities. Notably, all four providers expended more on occupational 
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The WorkAdvance Study 
Table 4.1 

Aggregate Gross Operating Costs (Y ears 1 to 3) 


Internal costs 

Year 1 

Year 2 

Year 3 

Total 

Per Scholas 

Total cost (S) 

584,665 

861,429 

573,344 

2,019,437 

Program components (%) 

Management 

7.8 

9.5 

19.2 

11.8 

Recruitment 

7.0 

17.0 

7.1 

11.3 

Job development and employer engagement 

14.4 

10.0 

14.0 

12.4 

Screening 

13.3 

10.4 

4.1 

9.5 

Preemployment services and support 

24.3 

16.9 

24.4 

21.2 

Occupational skills training 

33.2 

29.0 

19.0 

27.4 

Postemployment retention and advancement 

0.0 

7.2 

12.1 

6.5 

St. Nicks Alliance 

Total cost (S) 

388,240 

790,829 

688,514 

1,867,583 

Program components (%) 

Management 

8.7 

6.7 

12.3 

9.2 

Recruitment 

0.8 

6.6 

5.9 

5.2 

Job development and employer engagement 

5.8 

5.9 

5.2 

5.6 

Screening 

27.0 

12.8 

8.3 

14.1 

Preemployment services and support 

20.8 

23.1 

27.1 

24.1 

Occupational skills training 

24.9 

32.7 

22.9 

27.5 

Postemployment retention and advancement 

11.9 

12.2 

18.3 

14.4 

Madison Strateuies Groun 

Total cost ($) 

635,891 

976,794 

767,998 

2,380,682 

Program components (%) 

Management 

32.5 

14.5 

23.7 

22.3 

Recruitment 

6.0 

6.4 

8.7 

7.0 

Job development and employer engagement 

10.5 

6.9 

13.4 

10.0 

Screening 

13.0 

10.6 

8.5 

10.6 

Preemployment services and support 

10.2 

15.4 

15.8 

14.1 

Occupational skills training 

22.4 

37.4 

19.3 

27.5 

Postemployment retention and advancement 

5.5 

8.8 

10.6 

8.5 

Towards Emnlovment 

Total cost ($) 

628,469 

972,850 

1,027,074 

2,628,393 

Program components (%) 

Management 

30.4 

15.6 

10.0 

16.9 

Recruitment 

4.3 

5.1 

3.8 

4.4 

Job development and employer engagement 

28.0 

15.2 

14.4 

18.0 

Screening 

2.3 

3.4 

3.5 

3.2 

Preemployment services and support 

27.2 

22.9 

24.0 

24.4 

Occupational skills training 

7.3 

33.7 

38.4 

29.3 

Postemployment retention and advancement 

0.5 

4.0 

5.9 

3.9 


SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants 
Management System, and additional estimates by Per Scholas of uncaptured expenditures. 

NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) 
and operations in Youngstown, Ohio, have been excluded. Research-related costs have been excluded for all 
sites. 
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Box 4.1 


Definitions of the Cost Categories 

Management 

Costs associated with program administration, oversight, and staff supervision. (Costs attributable 
to management of the research are excluded.) 

Recruitment 

Cost of activities and materials necessary to attract eligible individuals to participate in 
WorkAdvance. (Costs attributable to the recruitment of the control group are excluded.) 

Job development and employer engagement 

Costs relevant to determining where jobs are available and where they should be sought for 
program participants; specific types of training necessary to fill those jobs; and the develop- 
ment of relationships with employers that can provide job opportunities to program partici- 
pants. Costs span preemployment and postemployment periods and encompass relationships 
with employers and trade-specific organizations. 

Screening 

Costs that result from determining whether individuals qualify for participation in a site’s 
WorkAdvance program (for example, testing, assessing work history, and so on). (Costs at- 
tributable to the screening of the control group are excluded.) 

Preemployment services and support 

Cost of providing program participants with general skills and other capabilities needed to 
obtain and keep a job (for example, soft skills training, interview coaching, financial assis- 
tance for employment- and training-related expenses, and so on). 

Occupational skills training 

Cost of providing program participants with the sector-specific skills necessary to perform 
jobs targeted by the respective WorkAdvance program. 

Postemployment retention and advancement 

Costs that arise from follow-up once program participants have obtained employment (for ex- 
ample, coaching participants through challenges and advancement opportunities in their em- 
ployment, interventions with the employers of participants when necessary, and so on). 
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The WorkAdvance Study 
Figure 4.1 

Percentage Distribution of Gross Operating Costs (Year 2) 



Management Recruitment Job development Screening Pre-employment Occupational skills Postemployment 


and employer 

services and 

training 

retention and 

engagement 

support 


advancement 

Program components 





SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants Management System, and additional estimates 
by Per Scholas of uncaptured expenditures. 


NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) and operations in Youngstown, Ohio, have 
been excluded. Research-related costs have been excluded for all sites. 



skills training — the program’s centerpiece — than on any other activity. Specifically, occupa- 
tional skills training accounted for roughly one-third of total operating costs at the four sites, 
ranging between 29 percent and 37 percent. In addition, preemployment services and support, 
which involved soft skills training and job interview coaching aimed at getting program partici- 
pants into the sorts of jobs for which they were being trained, accounted for 15 percent to 23 
percent of total gross expenditures at the four sites. Thus, over half of all expenditures at three 
of the four sites, and 46 percent at the remaining site (Per Scholas), were devoted to what might 
be termed “job readiness activities.” 

Per Scholas, the information technology (IT) training provider, which had operated a 
program similar to WorkAdvance for a number of years, is the only provider that spent well 
under a third of its budget on occupational skills training. Yet Per Scholas also enrolled a higher 
percentage of its participants in skills training, and those enrolled received more hours of train- 
ing. One plausible explanation for these differences is that Per Scholas is the only provider to 
have used its own employees exclusively to conduct training. It is conceivable that in-house 
training allowed Per Scholas to keep the per participant cost of occupational skills training rela- 
tively low, perhaps by avoiding various provider fees or allowing for greater flexibility among 
its staff members to multitask. The other three providers, in effect, absorbed the infrastructure 
costs of other institutions by paying outside organizations to provide training. In addition, Per 
Scholas had more years of experience in providing the type of training that it delivered in 
WorkAdvance than the other providers, which may have allowed it to use its resources more 
efficiently. 

As discussed in Chapter 1, a major objective of WorkAdvance was to help participants 
both maintain and advance in employment. In contrast to the expenditures on job readiness ac- 
tivities, however, postemployment retention and advancement never accounted for more than 
12 percent of total Year 2 gross expenditures. This probably stems from the fact that fewer 
WorkAdvance participants graduated from occupational skills training and were therefore po- 
tential recipients of postemployment retention and advancement aid during Year 2 than would 
have been the case in a true steady state. 6 Consequently, expenditures on this service are proba- 
bly smaller than would have occurred in an actual steady state. These numbers can be interpret- 
ed in the context of postemployment expenditures in Year 3, which probably overrepresents 
such expenditures relative to the steady state. 7 Even in Year 3, postemployment services never 


6 The relatively low number of graduates from occupational skills training can be attributed to the less ro- 
bust enrollment in Year 1 that is typical of program start-up. 

7 Appendix E indicates why postemployment costs in Year 2 are probably underrepresented and those in 
Year 3 are probably overrepresented. Overrepresentation of postemployment costs in Year 3 relative to Year 2 
implies that the percentage of total expenditures devoted to postemployment retention and advancement in 
Year 3 can be viewed as an upper bound. 
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account for more than 18 percent of total costs, so any underrepresentation in Year 2 is probably 
relatively minor. 

As discussed in Chapter 2, it took time to recruit a sufficient number of program partic- 
ipants. Even so, less than 10 percent of total provider costs were expended on recruitment dur- 
ing each of the three program years, 8 with an exception at Per Scholas (which made a large, 
one-time-only purchase of “outreach materials” in Year 2). As Table 4.1 indicates, recruitment 
costs at Per Scholas are similar to those at other providers in Years 1 and 3. 

Following recruitment, those who applied to WorkAdvance had to be screened by each 
provider to determine whether they qualified for the program and the study. The cost of this ef- 
fort was nontrivial at three of the four sites, where it accounted for between 10 percent and 13 
percent of total operating costs in Year 2. 9 In contrast, the Year 2 cost of screening made up on- 
ly 4.2 percent of costs at Towards Employment. 

As described Chapter 2, part of the WorkAdvance program model involved establishing 
relationships with employers, both to determine the sort of training that should be provided and 
to place and retain program participants in the types of jobs for which they were trained. To- 
wards Employment was the only provider to enlist the services of outside entities to perform 
this task, which may have contributed to the higher cost that it incurred for this activity relative 
to the other providers (see Figure 4.1). 

Although WorkAdvance is a fairly complex program, Table 4.1 indicates that the cost 
of managing it was kept relatively modest. The variation in the magnitude of management costs 
across providers is probably due to a degree of uncertainty on the part of the program directors 
as to which activities constitute management activities, which may have led to some incon- 
sistency across sites. Moreover, in measuring management costs, program directors attempted 
to net out time devoted to managing the research effort (as opposed to the program itself), and 
staff judgments about this probably also varied across sites. Still, it is evident that management 
costs, while not large, are a nontrivial fraction of total gross operating costs. 


8 Control group members, as well as those assigned to the WorkAdvance group, must be recruited before 
random assignment. Costs for recruiting control group members are excluded from Table 4.1, however, as well 
as from all subsequent tables. 

9 Because screening took place before random assignment, persons ultimately assigned to the control group 
as well as those assigned to the WorkAdvance group were screened. Costs for screening control group mem- 
bers are removed from the cost estimates. 
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Gross Costs per WorkAdvance Group Member 

Year 2 gross in-program costs per WorkAdvance group member (“per participant”) for 
each provider appear in Table 4.2. 10 These costs are broken down into the seven program com- 
ponents that were defined above. While they are reported as dollars per person instead of in per- 
centage temis, the pattern of expenditures is largely similar to the Year 2 cost infomiation con- 
veyed in Table 4.1 and Figure 4.1, so costs by program component are not discussed further in 
this section. However, Table 4.2 also breaks down costs into five budget components — salary, 
fringe benefits, overhead, consultants, and other program expenditures — that were not includ- 
ed in Table 4.1 and Figure 4.1. “Overhead” refers to expenses that support an organization or 
provide general resources or infrastructure for the program, such as rent, office supplies, equip- 
ment, and auditing and payroll services. “Consultants” are entities not directly employed by 
WorkAdvance organizations that provide services typically outlined in some sort of written 
agreement. In the context of WorkAdvance, consultants were most often used to provide skills 
training. “Other program expenditures” is a catch-all term for the remaining expenditures that 
are not directly related to personnel costs, which are captured by “salary” and “fringe benefits.” 

As previously noted, by far the most important expenditure engendered by WorkAd- 
vance was providing occupational skills training. Towards Employment covered this cost 
through tuition payments to schools, categorized as “other program expenditures,” while Per 
Scholas used its own employees as trainers and Madison Strategies and St. Nicks Alliance opt- 
ed to provide training by using subcontractors. Because Per Scholas depended so heavily on its 
own staff, its salary costs account for a much larger share of its total WorkAdvance budget than 
was the case across the other three providers (70 percent versus less than 40 percent). 11 

• Total gross costs per WorkAdvance group member in Year 2 are quite 
si mi lar among three of the four WorkAdvance sites, ranging between 
$5,200 and $5,750. The fourth site — St. Nicks Alliance — had gross 
costs in excess of $6,650 per WorkAdvance group member, which is 
$900 more than the next most expensive site. 

The cost of a similar training program and the cost of the same number of hours of 
training at a community college can put these WorkAdvance cost estimates into perspective. On 


10 As discussed in Appendix E, postemployment costs in Year 2 are likely to be understated relative to 
what they would be were WorkAdvance in a true steady state. Thus, in computing Y ear 2 costs per participant, 
the cost of postemployment services in Y ear 2 was replaced by the Y ear 3 cost of these services, which should 
bring the measure closer to an accurate approximation of the steady state. 

n Per Scholas’s fringe benefits were similar to those of the other providers because its fringe benefits as a 
percentage of salary were lower than those of the other providers (12 percent compared with 15 percent, 24 
percent, and 25 percent). 
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The WorkAdvance Study 
Table 4.2 

Steady-State Gross Operating Costs (per Participant) 


Budget Components 

Fringe Other Program 

Internal costs ($) Salary Benefits Overhead Consultants Expenditures Total 


Per Scholas 

Program components 

Management 446 

Recruitment 364 

Job development and employer engagement 467 

Screening 485 

Preemployment services and support 791 

Occupational skills training 1,003 

Postemployment retention and advancement 359 

Total 3,915 

St. Nicks Alliance 

Program components 

Management 285 

Recruitment 245 

Job development and employer engagement 249 

Screening 524 

Preemployment services and support 780 

Occupational skills training 24 

Postemployment retention and advancement 668 

Total 2,775 


52 

46 

0 

0 

544 

43 

81 

0 

480 

968 

55 

48 

0 

0 

570 

57 

50 

0 

0 

592 

93 

81 

0 

0 

965 

117 

400 

0 

135 

1,655 

52 

50 

0 

0 

461 

468 

756 

0 

616 

5,754 


70 

77 

0 

0 

431 

60 

76 

0 

45 

426 

61 

67 

0 

0 

377 

128 

146 

0 

22 

820 

191 

264 

0 

252 

1,487 

6 

374 

1,676 

22 

2,103 

151 

203 

0 

0 

1,022 

667 

1,206 

1,676 

342 

6,666 
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Table 4.2 (continued) 


Budget Components 


Internal costs ($) 

Salary 

Fringe 

Benefits 

Overhead 

Consultants 

Other Program 
Expenditures 

Total 

Madison Strategies Group 

Program components 

Management 

403 

53 

219 

77 

4 

757 

Recruitment 

210 

27 

50 

0 

49 

336 

Job development and employer engagement 

254 

33 

53 

19 

2 

362 

Screening 

415 

54 

82 

0 

3 

555 

Preemployment services and support 

484 

63 

119 

45 

94 

805 

Occupational skills training 

77 

10 

289 

1,267 

308 

1,951 

Postemployment retention and advancement 

222 

39 

89 

0 

87 

437 

Total 

2,066 

280 

901 

1,409 

547 

5,203 

Towards Employment 

Program components 

Management 

503 

119 

64 

119 

0 

804 

Recruitment 

103 

24 

21 

44 

69 

261 

Job development and employer engagement 

137 

32 

62 

553 

0 

784 

Screening 

104 

25 

14 

2 

32 

177 

Preemployment services and support 

665 

157 

94 

118 

145 

1,179 

Occupational skills training 

118 

28 

138 

0 

1,453 

1,737 

Postemployment retention and advancement 

219 

49 

51 

0 

0 

320 

Total 

1,849 

434 

445 

836 

1,699 

5,262 


SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants Management System, and 
additional estimates by Per Scholas of uncaptured expenditures. 

NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) and operations in Youngstown, 
Ohio, have been excluded. Research-related costs have been excluded for all sites. 



one hand, the cost per participant for the adult programs funded under the Job Training Partner- 
ship Act (JTPA) in the 1980s (and evaluated using an experimental evaluation similar to that for 
WorkAdvance) was about $4,500 in today’s dollars. 12 On the other hand, the annual expenses 
incurred by a community college that provides full-time students with approximately 450 hours 
of classroom instruction over the course of a year are generally above $10,000. 13 A typical full- 
time college student, like the typical WorkAdvance participant, also receives services in addi- 
tion to classroom instruction (such as assessment, advising, and help with course work). Per 
Scholas participants received on the order of 450 hours of classroom instruction, while Work- 
Advance participants at the other three sites received somewhat less. 

Relatively high gross costs per WorkAdvance group member at St. Nicks Alliance are 
reflected by St. Nicks being the most expensive provider in four of the seven core categories 
listed in Table 4.2 (screening, preemployment services and support, occupational skills training, 
and postemployment retention and advancement). These high costs appear to be attributable to 
multiple factors. For example, St. Nicks Alliance had substantially fewer participants than any 
of the other providers in Year 2 — less than two-thirds as many as Madison Strategies or To- 
wards Employment, and approximately four-fifths as many as Per Scholas — and thus operated 
at a smaller scale (which may have translated into excess capacity, for example if each career 
coach at St. Nicks served fewer participants than coaches at the other sites). Madison Strategies 
Group and Towards Employment also placed some participants directly into employment, re- 
sulting in lower costs than St. Nicks for any participants who did not receive occupational skills 
training or other WorkAdvance services. In addition, management costs per participant at St. 
Nicks were lower than those at the other three providers, possibly resulting in a less efficient 
operation. Finally, St. Nicks Alliance operated in a high-cost environment (New York City). 
These factors are discussed in greater detail in Appendix A. 

Net Costs per WorkAdvance Group Member 

Table 4.3 displays the net costs for WorkAdvance. Gross costs per WorkAdvance 
group member are shown in the table’s first column, and gross costs associated with control 
group members are shown in the second. Estimates of WorkAdvance’s net costs per person are 
computed by subtracting the second column from the first and are reported in the third column. 
Table 4.3 also shows “in-program” and “out-of-program” costs separately for three categories 
of activities: preemployment services and support, occupational skills training, and post- 


12 Bloom et al. (1997), Table 1. Costs were originally estimated in 1987-1989 dollars; the $4,500 estimate 
reflects inflation adjustment into 2012-2013 dollars. 

13 See Appendix A for information on data sources. 


107 



The WorkAdvance Study 
Table 4.3 


Gross and Net In-Program and Out-of-Program Costs (per Individual) 


Per person costs ($) 

Gross Costs 

WA 

C 

Net Costs 

Per Scholas 

Program components 

Management 

544 

0 

544 

Recruitment 

968 

0 

968 

Job development and employer engagement 

570 

0 

570 

Screening 

592 

0 

592 

Preemployment services and support 

In-program 

965 

0 

965 

Out-of-program 

34 

34 

0 

Occupational skills training 

In-program 

1,655 

0 

1,655 

Out-of-program 

391 

2,626 

-2,234 

Postemployment retention and advancement 

In-program 

461 

0 

461 

Out-of-program 

35 

79 

-44 

Total costs 

6,214 

2,739 

3,476 

St. Nicks Alliance 

Program components 

Management 

431 

0 

431 

Recruitment 

426 

0 

426 

Job development and employer engagement 

377 

0 

377 

Screening 

820 

0 

820 

Preemployment services and support 

In-program 

1,487 

0 

1,487 

Out-of-program 

39 

39 

0 

Occupational skills training 

In-program 

2,103 

0 

2,103 

Out-of-program 

898 

1,625 

-727 

Postemployment retention and advancement 

In-program 

1,022 

0 

1,022 

Out-of-program 

93 

164 

-71 

Total costs 

7,696 

1,828 

5,868 


(continued) 
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Table 4.3 (continued) 


Per person costs ($) 

Gross Costs 

WA 

C 

Net Costs 

Madison Strategies Group 

Program components 

Management 

757 

0 

757 

Recruitment 

336 

0 

336 

Job development and employer engagement 

362 

0 

362 

Screening 

555 

0 

555 

Preemployment services and support 

In-program 

805 

0 

805 

Out-of-program 

28 

39 

-11 

Occupational skills training 

In-program 

1,951 

0 

1,951 

Out-of-program 

781 

982 

-201 

Postemployment retention and advancement 

In-program 

437 

0 

437 

Out-of-program 

36 

68 

-32 

Total costs 

6,048 

1,089 

4,959 

Towards Employment 

Program components 

Management 

804 

0 

804 

Recruitment 

261 

0 

261 

Job development and employer engagement 

784 

0 

784 

Screening 

177 

0 

177 

Preemployment services and support 

In-program 

1,179 

0 

1,179 

Out-of-program 

26 

30 

-4 

Occupational skills training 

In-program 

1,737 

0 

1,737 

Out-of-program 

1,145 

1,522 

-378 

Postemployment retention and advancement 

In-program 

320 

0 

320 

Out-of-program 

11 

37 

-26 

Total costs 

6,443 

1,589 

4,855 


SOURCES: In-program costs are derived from financial reports submitted to The Mayor's Fund to 
Advance New York City through the Grants Management System and from additional Per Scholas 
estimates of uncaptured expenditures. Out-of-program costs are based on estimates developed by MDRC 
using figures from: (1) the Integrated Postsecondary Education Data System, hosted by the U.S. 
Department of Education's National Center for Education Studies; (2) U.S. Department of Labor 
Employment and Training Administration Program Year 2013 Final Allotments and employment 
outcomes; and (3) official budgets issued to WorkAdvance Social Innovation Fund subgrantees. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Costs attributable to Compass (Towards Employment's partner in the administration of 
WorkAdvance) and operations in Youngstown, Ohio, have been excluded. Research-related costs have 
been excluded for all sites. 
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employment retention and advancement. In-program expenditures for these activities were in- 
curred only by the WorkAdvance group and were paid from the program’s budget for Work- 
Advance. 14 In contrast, both the WorkAdvance group and the control group incurred out-of- 
program costs, which were paid by sources other than the WorkAdvance budget. For example, 
costs may have been covered by community colleges, for-profit training organizations, or state 
employment services, or paid out of pocket by those participating in the relevant activities. All 
costs incurred in the other four categories listed in the table were in-program. In-program costs 
also appear in Table 4.2, but out-of-program costs appear only in Table 4.3. 

The key finding from Table 4.3 is that net costs are smaller than the gross in-program 
operating cost figures that appear in Table 4.2, as expected; but with the rather striking excep- 
tion of Per Scholas, they are not much smaller. This is partly because out-of-program occupa- 
tional skills training costs for control group members at sites other than Per Scholas were sub- 
stantially offset by out-of-program costs for the WorkAdvance groups at the respective sites. As 
a consequence, net costs for three of the four sites are relatively similar to gross costs, falling 
only 5 percent to 12 percent below gross costs. Per Scholas, in contrast, had net costs that fell 40 
percent below its gross costs. In absolute tenns, net costs per individual for Per Scholas are 
around $3,500, which is over $1,300 lower than for the next lowest site (Towards Employ- 
ment). 


• Per Scholas had substantially lower net costs than any other provider, in 
part because its control group members incurred considerably more 
out-of-program training costs than those at any other site. 

As discussed in greater detail below, Per Scholas’ relatively low net costs arise because 
the cost of out-of-program occupational skills training at Per Scholas is much larger for the con- 
trol group than for the WorkAdvance group. Indeed, Table 4.3 shows that out-of-program costs 
for control group members at Per Scholas are substantially higher than they are at the other 
three sites and out-of-program costs for the members of the WorkAdvance group are substan- 
tially lower. 

Table 4.3 indicates that two of the categories of out-of-program costs — pre- 
employment services and support, and postemployment retention and advancement — are of 
relatively minor importance. Their magnitude is small for both the WorkAdvance group and the 
control group at each site. Because the cost for the control group is partially offset by the cost 
for the WorkAdvance group, their effect on net costs is also slight. 


14 Management, recruitment, and screening costs were incurred for the control group, but these costs were 
entirely for research purposes — they would not have existed in the absence of an evaluation. Therefore, they 
are not included in the costs reported in Table 4.3. 
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Out-of-program preemployment service and support costs were incurred when mem- 
bers of the WorkAdvance and control groups obtained a job through employment agencies or 
community-based organizations (CBOs), and, as indicated by Table 3.4, the fraction who ob- 
tained a job in this way was relatively small (usually under one-quarter) and not much larger for 
control group members than for the WorkAdvance group members. Moreover, the expenditures 
that are incurred when jobs are obtained through employment agencies are often fairly small. 
State employment offices, for example, rely heavily on computerized lists and other measures 
that help bring costs down. 

Out-of-program postemployment retention and advancement costs resulted from mem- 
bers of the WorkAdvance and control groups receiving help from an employment agency or 
CBO to retain a job. Relatively few did: Never more than 5 percent of the control groups and an 
even smaller percentage of the WorkAdvance groups received this sort of assistance. 

Costs associated with out-of-program occupational skills training arise from courses 
taken at high schools, community colleges, four-year colleges, for-profit providers, and non- 
profit organizations. Of these, community colleges and for-profit providers account for most of 
the measured cost of out-of-program training received by members of the WorkAdvance re- 
search sample. Combined, these two categories account for 84 percent of the control group 
training costs at St. Nicks Alliance, 95 percent of such costs at Madison Strategies Group, 86 
percent of such costs at Towards Employment, and 66 percent of such costs at Per Scholas. 
Four-year colleges also played an important role among control group members at Per Scholas, 
accounting for 20 percent of the out-of-program training costs. 

Why Are Per Scholas’s Net Costs So Low? 

The costs of out-of-program training reported in Table 4.3 are a reflection of several 
factors: the percentage of the WorkAdvance and control group members that enroll in courses; 
whether enrollees are full-time or part-time students; the number of months that those who en- 
roll remain in class; and the per student expenses borne by the local educational institutions in 
which they enroll (for example, the cost of providing instruction, institutional support, and stu- 
dent services). Together, these factors contributed to Per Scholas’s relatively low net costs per 
participant. 

The percentage of each group that enrolls is of particular importance. Table 3.2 indi- 
cates that the percentages of WorkAdvance group members who enrolled at community colleg- 
es and at for-profit training providers were lower at Per Scholas than at the other three sites (of- 
ten considerably so), perhaps because a higher proportion of Per Scholas participants took skills 
training provided by WorkAdvance and this training was longer and more intense than at the 
other three providers. It is possible that as a consequence, more program participants at Per 
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Scholas than at the other sites became convinced that they had received the training they needed 
to improve their lives and therefore did not seek it elsewhere. 

In contrast to the Per Scholas WorkAdvance group, Per Scholas control group mem- 
bers, when taken as a whole, enrolled in occupational skills training and college courses at a 
higher rate than at any of the other sites. Furthermore, those who enrolled generally spent more 
hours in training. This may be attributable to the lower age and the higher level of education of 
Per Scholas control group members relative to control group members at the other sites (Table 
1 .4). Although the percentages of control group members enrolling in community colleges and 
at for-profit providers were not the highest at Per Scholas, they were relatively high among the 
four sites. Moreover, Per Scholas control group members were the only group among which 
more than a minuscule percentage enrolled in four-year colleges. The expenditures resulting 
from this enrollment in four-year colleges account for roughly a third of the differences between 
the control group out-of-program costs at Per Scholas and those at Madison Strategies and To- 
wards Employment, and for almost half the difference between Per Scholas and St. Nicks Alli- 
ance. Most of the remaining differences between Per Scholas out-of-program control group 
costs and the corresponding costs at the other three sites can be attributed to expenses resulting 
from courses taken at community colleges and for-profit providers. 

As shown in Table 3.2, the differences in favor of the control group (versus the Work- 
Advance group) in enrollment at community colleges and for-profit training providers were 
much larger at Per Scholas than at the other three sites. In addition, results from the Year 2 Sur- 
vey indicate that among those taking training at community colleges or for-profit providers, Per 
Scholas control group members enrolled for more months than their counterpart WorkAdvance 
group members, a gap in outcomes that was considerably larger than the corresponding gaps at 
the other three sites. These gaps are reflected in the difference between the out-of-program oc- 
cupational skills training costs that were incurred by Per Scholas control group members and by 
Per Scholas WorkAdvance group members — a difference that is much larger than at any of the 
other three sites. 

Per student educational expenditures — especially at community colleges — were 
somewhat larger in the Bronx, where Per Scholas is located, than at the locations of the other 
three WorkAdvance programs. This tended to increase out-of-program educational costs for 
control group members in the area relative to their counterpart, the Per Scholas WorkAdvance 
group members, thereby decreasing Per Scholas’s estimated net cost. However, when average 
per student expenditures at the other three sites are substituted for per student expenditures in 
the Bronx, Per Scholas’ net costs increase by only $286, suggesting that higher local education- 
al costs in the Bronx account for little of Per Scholas’s relatively small estimated net costs. 
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Conclusion 


One key finding from the cost analysis is that gross program operating costs per WorkAdvance 
group member (which include only in-program costs) ranged from $5,200 to $5,800 at three of 
the four sites, and were $900 higher at the fourth site (St. Nicks Alliance). The narrow range at 
three of the sites suggests that similar WorkAdvance programs run elsewhere are likely to incur 
costs within this general range, and the higher costs at the fourth site might be viewed as a po- 
tential upper bound. Moreover, considering that the programs at all four sites allocated roughly 
half their operating expenditures to providing preemployment activities and occupational skills 
training, it would be reasonable for any programs replicating WorkAdvance as it is implement- 
ed here to incorporate a similar distribution of costs into their budgets. 

Net cost figures are the extra cost society spends letting eligible individuals attend 
WorkAdvance rather than what they would have done otherwise, in the absence of WorkAd- 
vance. Because WorkAdvance encompasses activities that individuals targeted by the program 
would have engaged in on their own — as the control group members did — net costs (which 
include both in-program and out-of-program costs) are smaller than gross in-program operating 
costs. However, with the exception of Per Scholas, they do not appear to be much smaller. The 
differences between gross in-program operating costs and net costs are almost entirely attributa- 
ble to control group members enrolling in community colleges and for-profit institutions, and, 
in the case of Per Scholas, in four-year colleges. Members of the WorkAdvance group also en- 
rolled in these institutions, but to a lesser extent than control group members, especially at Per 
Scholas. 


The next chapter examines the economic effects of the difference in services received 
by WorkAdvance and control group members (as well as the attendant differences in cost). 
Thus, the next chapter’s focus is on WorkAdvance’s effects on such outcomes as employment 
in the targeted sector and total earnings and income. 
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Chapter 5 

Economic Impacts of WorkAdvance 

Chapter Highlights 

• The employment and earnings effects of the WorkAdvance programs varied 
across the providers, but overall, the findings show that well-implemented 
sector-based training programs can improve participants’ economic out- 
comes. 

• Per Scholas produced large impacts on employment and earnings throughout 
most of the follow-up period. These impacts were substantial enough to 
translate into positive impacts on income, public benefit receipt, and life sat- 
isfaction. 

• The WorkAdvance program at St. Nicks Alliance had little to no effect on 
employment and earnings, and only a few impacts on secondary economic 
outcomes. 

• Towards Employment increased employment and earnings, primarily late in 
the follow-up period. The program produced impacts on a few measures of 
advancement but had little effect on other secondary outcomes. 

• The WorkAdvance program at Madison Strategies Group increased earnings 
late in the follow-up period but did not have a systematic detectable effect on 
employment. The program produced impacts on several secondary outcomes, 
including nonfinancial advancement measures and receiving employer- 
provided benefits. 


Introduction 

Chapter 3 shows large differentials in service receipt between WorkAdvance group members 
and control group members at all four sites. Using data from the Year 2 Survey and unemploy- 
ment insurance wage records, this chapter discusses whether these large differentials in service 
receipt led to impacts on economic outcomes. As in Chapters 2 and 3, the results in this chapter 
are discussed by site within domains. 1 The first domain — employment and earnings — is con- 

'ln the original Subgrantee Evaluation Plan, the intention was to analyze impacts by sector. Because most 
sites focused on a unique sector, the only difference between analyzing impacts at the site level and at the sec- 
tor level is in Ohio. But since the Compass site in Ohio was eliminated from the study, the statistical power was 
insufficient to reliably estimate impacts by sector. Therefore, a decision was made in the analysis planning 
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sidered primary. This domain includes employment and earnings, as measured by both data 
sources, as well as various exploratory measures from the Year 2 Survey. The other domains 
include income, life satisfaction, and material hardship, and are considered secondary. 

Data Sources and Samples 

The findings on employment, earnings, and other economic outcomes presented in this 
chapter come from two data sources: the WorkAdvance Year 2 Survey and unemployment in- 
surance wage records. While both data sources provide valuable infonnation, there are some 
important differences and trade-offs between the two sources. Administrative data are available 
for the full sample, but as discussed in Chapter 1, at least 10 percent of jobs are not covered and 
these records cover employment only within the state. Survey data cover all jobs, but survey 
participants often do not recall the details of employment accurately, and approximately 20 per- 
cent of the sample did not respond to the survey. Taking into account both data sources enables 
the most reliable, composite assessment of impacts. For the most part, the findings are con- 
sistent across the two data sources. 2 

The WorkAdvance Year 2 Survey asked sample members about their participation in 
workforce services (these findings are presented in Chapter 3), as well as their employment and 
earnings history since study entry. Sample members were interviewed for the survey between 
18 and 30 months after they entered the study; the average sample member was interviewed 22 
months after entering the study. Overall, the survey achieved an 80 percent response rate; the 
response rate for the WorkAdvance group (83 percent) was slightly higher than for the control 
group (77 percent). 3 Most survey-based measures presented in this chapter cover the full follow- 
up period for each respondent (from the date of study entry through the date of the survey inter- 
view), regardless of the month the respondent was interviewed. Measures that are reported in 
specific time intervals (for example, average number of months in current or most recent job) 
cover only the first 18 months relative to each respondent’s month of random assignment. This 
18-month coverage period is referred to as the common follow-up period. 


stage to estimate impacts at the site level. Ultimately, this affects only the Ohio site because every other site 
focused on a single and different sector (broadly defined). Findings are shown by site based on the results of a 
power analysis conducted during the analysis planning phase. Additional information on the power analysis 
can be found in Appendix A. Pooled impacts and impacts by cohort and subgroup are shown in Chapter 6. 

‘While the results are generally consistent between the two data sources, in some cases the results are sta- 
tistically significant for the larger administrative records sample but not for the survey. For example, this pat- 
tern occurs at the Towards Employment site for measures of employment. See Appendix A for the survey re- 
sponse analysis and more details on the comparison between survey-based and unemployment insurance-based 
employment findings. 

3 The response rates by site are Per Scholas, 80 percent; St. Nicks Alliance, 80 percent; Madison Strategies 
Group, 80 percent; and Towards Employment, 81 percent. Appendix A contains more details on the Year 2 
Survey response rates. 
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Findings based on unemployment insurance records data are presented for the full re- 
search sample. Nine quarters of follow-up data are available for all sample members. 4 Unem- 
ployment insurance records are available quarterly, and findings are presented relative to each 
sample member’s quarter of random assigmnent. For example, “Year 1” refers to the first four 
quarters following each sample member’s quarter of random assigmnent. 

The Expected Impacts of the WorkAdvance Programs 

As mentioned in Chapter 1, only one in five sample members were employed at the 
time they entered the study. This was expected given that the services offered by the program 
— career readiness and occupational skills training — required a substantial time co mmit ment. 
The WorkAdvance model was designed to increase employment and earnings by offering these 
services along with job placement and postemployment services that would help participants 
obtain jobs with opportunities for career advancement. Based on the model, it was hypothesized 
that the WorkAdvance programs would first increase employment, primarily in the targeted sec- 
tor, and then increase earnings. 

As noted earlier, two of the providers — Madison Strategies Group and Towards Em- 
ployment — initially offered a placement-first track in addition to the training-first track. For 
participants in the placement-first track, 5 it was thought that employment and earnings impacts 
would begin to emerge shortly after study entry, as participants did not engage in occupational 
skills training and went directly into job search after career readiness training. However, be- 
cause these participants did not go to skills training (at least initially), they may not have re- 
ceived as intensive a set of services as did the participants in the training-first track. The treat- 
ment contrast for this group may not have been enough for them to obtain better jobs than their 
control group counterparts. 6 

For participants in the training-first track (the majority of participants), it was expected 
that WorkAdvance group members would have lower employment rates, and therefore earn- 
ings, than the control group in the first few months following study entry owing to the oppor- 
tunity cost of training. It was thought that not all control group members would engage in occu- 
pational skills training on their own due to the often high costs of training and the limited avail- 


4 Ten quarters of data were collected altogether. Quarter 1 was the quarter of random assignment and is 
therefore not included in the follow-up data. 

5 Sample members were not randomly assigned into one of the two tracks, and therefore the exact number 
of sample members in the placement-first track is not known. 

6 The placement-first track was phased out for the most part in fall 2012 due to concerns about participants 
getting “stuck” in dead-end jobs. However, having some initial placements may have helped these providers 
(each of which was dealing with a new sector or a new location) to establish relationships with employers more 
quickly than they would have had they started with a pure training program. 
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ability of public funding for it. 7 Instead, most control group members would probably seek 
more immediate employment. Therefore, only after WorkAdvance group members completed 
occupational skills training and looked for and obtained a job would employment and earnings 
for that group begin to increase. 

Past studies of workforce programs have shown it can take up to two or even three 
years before economic impacts begin to emerge. 8 After accounting for the time spent in occupa- 
tional skills training, only a little over one year of follow-up data are available. 9 It was projected 
that in the Year 2 Survey data (which include infonnation on the receipt of postemployment 
services and on advancement outcomes), only initial job placements would be seen for most 
sample members, and that any advancement gains stemming from postemployment services (a 
key and distinguishing component of the WorkAdvance model) would not yet have material- 
ized. The unemployment insurance data presented in this report cover a longer follow-up period 
than the Year 2 Survey data (nine quarters), and so it was thought that these data might capture 
subsequent jobs and initial advancement gains. Plans are also being made to collect longer-term 
data beyond the follow-up period covered by this report to capture future advancement. 

The Control Group Benchmark 

Control group members were not eligible to receive WorkAdvance services, and there- 
fore their outcomes can be used as a benchmark against which to measure the impacts of the 
WorkAdvance programs. Employment and earnings outcomes for the control group represent 
what would have happened in the absence of WorkAdvance. Control group members were free 
to seek out any non-WorkAdvance employment services and training opportunities in their local 
communities. 

• Employment rates reported in both the Year 2 Survey and in unem- 
ployment insurance wage records are high (between 84 percent and 95 
percent) for the control groups at all four sites. Earnings levels for the 
control group increased over time. 

The amount of screening required by applicants may have contributed to their high em- 
ployment rates. 10 Madison Strategies Group control group members had the highest employ- 
ment rates in both the Year 2 Survey (95 percent) and in unemployment insurance wage records 

’National Skills Coalition (2011). 

8 Card, Kluve, and Weber (2010); D’Amico (2006). 

9 At Madison Strategies Group and Towards Employment, control group members ended primary skills 
training significantly earlier (between two and three months earlier) than WorkAdvance group members. 

10 As discussed previously, only one in five applicants made it through the intake process. The control 
group appeared to be motivated and fairly likely to seek out other, similar services on their own (see Chapter 
3). 
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(93 percent). During much of the follow-up period there was an economic boom in the Tulsa 
region related to the oil and gas sector, which made it very easy for control group members to 
find employment and which provided a higher bar to clear for the program to produce impacts. 
Among the control groups, Towards Employment had the lowest employment rate as reported 
by the survey (88 percent), and St. Nicks Alliance had the lowest employment rate from unem- 
ployment insurance wage records (84 percent). From Year 1 to Year 2, employment rates for 
control group members increased somewhat at two sites and decreased somewhat at the other 
two sites. However, earnings levels increased at all four sites during the same time period, sug- 
gesting that some control group members who stayed employed were able to increase their 
earnings. Control group members at Madison Strategies Group had the highest average earnings 
in Year 2 ($14,804) and the highest rate of control group members earning $20,000 or more (31 
percent). 


Impacts on Employment and Earnings 

This section presents employment and earnings impacts separately for the two data sources: the 
Year 2 Survey and unemployment insurance wage records. 11 

Employment and Earnings Impacts from the Year 2 Survey 

Table 5.1 presents impacts on employment and wages as measured by the Year 2 Sur- 
vey. Survey respondents were allowed to report up to eight jobs that they had held since random 
assignment. The first four measures in Table 5.1 on employment status and the amount of time 
employed look across all the jobs reported by survey respondents. The remaining measures in 
the table represent only survey respondents’ current or most recent jobs at the time of the survey 
interview. 12 Most of the measures in the table cover the entire follow-up period, except for the 


"Appendix F includes supplementary exhibits showing employment and earnings impacts. All estimates 
presented in this chapter were regression adjusted using ordinary least squares, controlling for pre-random as- 
signment characteristics of sample members (referred to as covariates). The covariates differ somewhat by site, 
but in general include random assignment year, gender, race/ethnicity, age, education attainment, number of 
children, whether born in the state, previous license or certification in the targeted industry, employment status 
at baseline, monthly family income, and employment and earnings history before random assignment. Addi- 
tional information on co variate selection can be found in Appendix A. 

12 For sample members who were employed at the time of the survey interview, these measures capture the 
job in which they were working at that time (their “current job”). If sample members were working two or 
more jobs at the time of the survey interview, the job where they worked the most hours was considered their 
“current job.” For sample members who had worked since random assignment but were not working at the 
time of the survey interview, these measures capture the job that ended most recently before the time of the 
survey interview (their “most recent job”). Impacts on characteristics of the current job are shown in Appendix 
Table F.l. 
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The WorkAdvance Study 
Table 5.1 


Year 2 Impacts on Employment and Wages, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

All jobs 

Ever employed 

94.5 

90.1 

4.4 * 

90.0 

90.8 

-0.8 

96.1 

94.6 

1.5 

89.2 

87.5 

1.7 

Currently employed 

74.4 

64.4 

10.0 *** 

65.3 

65.4 

-0.1 

77.6 

72.0 

5.6 

64.5 

58.2 

6.3 

Percentage of months employed 11 

57.0 

53.2 

3.8 

53.7 

52.7 

0.9 

65.7 

67.6 

-2.0 

52.5 

57.5 

* 

O 

ITT 

i 

Employed 6 or more consecutive months 1 

73.1 

65.7 

7.5 * 

68.7 

61.6 

7.1 

77.1 

79.7 

-2.6 

62.1 

66.4 

-4.3 

Current or most recent job 

Employed in targeted sector 

61.1 

20.4 

40 7 *** 

31.8 

19.9 

l \ 9 *** 

64.6 

48.2 

16.5 *** 

50.1 

33.6 

*** 

Earnings 

Average total earnings per week ($) 

561 

463 

*** 

520 

504 

16 

551 

525 

27 

379 

360 

19 

Average hourly wage ($) 

15.23 

12.85 

2 38 *** 

13.07 

13.34 

-0.27 

12.47 

12.01 

0.46 

10.11 

9.78 

0.33 

Employed and hourly wage above $12.00 

63.1 

46.8 

16.3 *** 

53.3 

50.9 

2.4 

53.2 

41.9 

H 2 *** 

28.5 

26.7 

1.8 

Employed and hourly wage above $15.00 

46.0 

29.8 

16.2 *** 

27.7 

26.9 

0.8 

23.6 

17.0 

6.6 * 

8.7 

11.5 

-2.8 

Hours 

Average hours worked per week (#) 

34.3 

33.2 

1.1 

36.0 

34.1 

1.9 

42.0 

40.5 

1.4 

33.1 

31.6 

1.5 

Employed full time b 

68.1 

62.6 

5.5 

70.1 

66.3 

3.8 

82.3 

79.4 

2.9 

60.4 

53.7 

6.7 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

Sample sizes may vary because of missing values. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Measures reported in time intervals cover only the first 18 months following each sample member's month of random assignment (the common 
follow-up period). 

b Full-time employment is defined as working 35 hours or more in a week. 



percentage of months employed and whether the respondent was employed for six or more con- 
secutive months; for consistency, these two measures cover only the common follow-up period 
— the first 18 months following each sample member’s month of random assigmnent. 

Targeted Sector Employment from the Year 2 Survey 

• At all four providers, WorkAdvance increased employment in the tar- 
geted sector relative to the control groups. 

Among the providers, Per Scholas produced by far the largest impact on employment in 
the targeted sector: 61 percent of WorkAdvance group members reported working in the infor- 
mation technology (IT) sector at their current or most recent job, a striking 41 percentage points 
higher than the control group. St. Nicks Alliance had the lowest rate of WorkAdvance group 
members who reported working in the targeted sector (environmental remediation) at 32 per- 
cent, but the impact of 12 percentage points is still statistically significant. 13 Almost two-thirds 
of WorkAdvance group members at Madison Strategies Group and half of WorkAdvance group 
members at Towards Employment were employed in the targeted sector, compared with 48 
percent and 34 percent, respectively, of control group members (Table 5.1). The high control 
group level at these two sites might be due to lower barriers to entry into those sectors, com- 
pared with the barriers to entry into the IT and environmental remediation sectors. 

While it is impressive that all providers were able to move a significant proportion of 
participants into jobs in the targeted sectors, this represents a necessary but not sufficient condi- 
tion for impacts on overall employment, earnings, and advancement. For these effects to have 
occurred, wages and benefits at jobs in the targeted sector would have to be better than at jobs 
outside the targeted sector. The rest of this chapter shows whether there are impacts on em- 
ployment, earnings, job characteristics (such as schedules and job satisfaction), and advance- 
ment, and the next chapter provides an indication of whether targeted sector jobs have better 
characteristics (wages, hours, and so on) than jobs outside the targeted sector. 

Overall Employment and Current Employment from the Year 2 Survey 

The first two measures in Table 5.1 show the percentage of survey respondents ever 
employed since random assignment and the percentage of respondents employed at the time of 
the survey interview (this is, on average, Month 22). The latter provides an indication of em- 
ployment retention. 


13 It is important to note that the measure used here indicates whether employment in the current or most 
recent job is in the targeted sector. Because St. Nicks often places individuals in short-term project work and 
individuals might move in and out of the sector, it is perhaps more likely that this measure might underestimate 
targeted sector employment at that site. 
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• Only the WorkAdvance program at Per Scholas produced a statistically 
significant increase in employment since random assignment and in cur- 
rent employment at the time of the Year 2 Survey, relative to the control 
group. Towards Employment and Madison Strategies Group produced 
positive, although not statistically significant, impacts on current em- 
ployment. 

Table 5.1 shows that around 90 percent of control group members at all four sites were 
ever employed since random assigmnent. Despite the high control group employment rate, Per 
Scholas WorkAdvance group members were 4 percentage points more likely to have ever 
worked than control group members. Almost 95 percent of Per Scholas WorkAdvance group 
members worked since random assigmnent compared with 90 percent of control group mem- 
bers. Per Scholas was the only provider to produce a statistically significant impact on overall 
employment measured by the survey. 

At all four sites, around 20 percent to 25 percent of respondents in both research 
groups lost a job following study entry and did not obtain a new job by the time of the survey 
interview (measured as the difference between “ever employed” and “currently employed”). 
Per Scholas was the only provider to produce an impact on the current employment measure 
— 74 percent of WorkAdvance group members at the site were currently employed at the 
time of the survey interview, an increase of 10 percentage points over the control group. The 
WorkAdvance programs at both Madison Strategies Group and Towards Employment pro- 
duced increases in current employment relative to the control group, and while the estimated 
impacts were not statistically significant, both are just above the cutoff for statistical signifi- 
cance. 14 


Earnings and Hours from the Year 2 Survey 

• The WorkAdvance program at Per Scholas produced impacts on hourly 
wages and weekly earnings. At Madison Strategies Group, the Work- 
Advance program increased the percentage of respondents working and 
earning more than $12 per hour. WorkAdvance had no effect on wages 
or earnings in the current or most recent job at the other two sites. 

The estimated weekly earnings for Per Scholas WorkAdvance group members were 
$561, a statistically significant increase of $98, or 21 percent higher than the control group av- 
erage. WorkAdvance group members at the site also reported significantly higher hourly wages 


14 The cutoff for statistical significance used in this report is 0.100. The p-value on current employment at 
Madison Strategies Group is 0.122 and at Towards Employment is 0.109. With slightly larger sample sizes, 
both of these estimated impacts would have been statistically significant, assuming constant effect sizes. 
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($15.23) compared with the control group ($12.85) and were 16 percentage points more likely 
than control group members to be employed with a wage above $15 per hour (Table 5.1). 

Madison Strategies Group did not have an effect on weekly earnings or average hourly 
wages, 15 but the provider did increase the percentage of respondents employed since random 
assigmnent who earned above $12 per hour and who earned above $15 per hour. Around 53 
percent of WorkAdvance group members at Madison Strategies Group found a job with a wage 
above $12 per hour, compared with 42 percent of control group members, a statistically signifi- 
cant increase of 1 1 percentage points. 

At Towards Employment and St. Nicks Alliance, WorkAdvance did not produce statis- 
tically significant impacts on wages or weekly earnings. Average hourly wages and weekly 
earnings were lowest for both research groups at Towards Employment, perhaps reflecting the 
economic conditions in the local labor market of the site — Cleveland, Ohio — which has ex- 
perienced a prolonged period of deindustrialization. WorkAdvance at Towards Employment did 
increase the likelihood of working at wages above $10 per hour by 4 percentage points (not 
shown), but this increase was not statistically significant. WorkAdvance group members at To- 
wards Employment earned an average of $10.11 per hour at their current or most recent job, 
while control group members earned an average of $9.78. 

• WorkAdvance did not produce statistically significant impacts on work 
hours at any of the sites. 

The majority of respondents at all four sites worked full time (at least 35 hours per 
week). Respondents at Madison Strategies Group worked the most hours per week across the 
sites — an average of 42 hours per week for WorkAdvance group members and an average of 
41 hours per week for control group members (Table 5.1). 

Employment and Earnings Impacts from Unemployment Insurance 

Wage Data 

This section presents employment and earnings impacts based on unemployment insur- 
ance wage data. These findings differ slightly from the findings from the Year 2 Survey because 
of differences in samples and jobs covered, but overall, the findings are consistent. Table 5.2 
presents impacts over several time periods. The first set of measures captures the full follow-up 
period — Quarters 2 to 10. Because, as discussed earlier, part of the follow-up period includes 


15 When measured based on current or most recent jobs, which was the prespecified outcome of primary 
interest in this report, these effects are not statistically significant. However, looking only at current jobs (held 
at the time of the survey interview), the WorkAdvance program at Madison Strategies Group did increase 
weekly earnings by a statistically significant amount (as shown in Appendix Table F.l). 
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The WorkAdvance Study 
Table 5.2 

Quarters 2 to 10 Impacts on Unemployment Insurance-Covered Employment and Earnings, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Quarters 2-10 

Ever employed (%) 

89.0 

86.1 

2.9 

87.6 

83.5 

4.1 

92.8 

92.8 

0.0 

92.5 

86.6 

5 g *** 

Average quarterly employment (%) 

61.5 

57.4 

4.2 * 

56.7 

55.2 

1.5 

69.9 

67.5 

2.4 

65.3 

62.2 

3.1 

Earnings ($) 

32,470 27,284 

5,186 ** 

27,782 

27,982 

-201 

33,759 

31,207 

2,552 

26,036 24,227 

1,809 

Year 1 (Quarters 2-5) 

Ever employed (%) 

77.1 

71.0 

6.1 * 

78.2 

70.4 

7 y ** 

89.3 

86.5 

2.8 

83.6 

78.8 

4.8 * 

Average quarterly employment (%) 

53.3 

51.1 

2.2 

52.8 

50.6 

2.2 

72.6 

68.8 

3.8 

61.9 

61.4 

0.5 

Employed in all 4 quarters (%) 

24.5 

28.0 

-3.5 

20.8 

27.7 

-6.9 * 

50.1 

48.8 

1.3 

38.1 

42.4 

-4.3 

Earnings ($) 

8,868 

8,718 

150 

9,395 

9,648 

-253 

13,261 

12,933 

328 

9,495 

9,483 

12 

Earned $20,000 or more (%) 

13.2 

13.5 

-0.3 

13.7 

17.7 

-4.0 

26.2 

23.4 

2.8 

14.2 

17.0 

- 2.8 

Year 2 (Quarters 6-9) 

Ever employed (%) 

82.4 

76.4 

6.0 ** 

74.3 

72.2 

2.1 

80.3 

82.8 

-2.5 

79.1 

73.7 

5.4 * 

Average quarterly employment (%) 

67.5 

61.8 

5.7 * 

59.8 

57.8 

2.0 

68.4 

67.6 

0.8 

67.6 

62.5 

5.1 * 

Employed in all 4 quarters (%) 

50.4 

43.4 

7.0 * 

44.7 

39.2 

5.5 

55.3 

52.5 

2.8 

53.0 

48.1 

4.9 

Earnings ($) 

18,217 

14,471 

3,747 *** 

14,420 

14,229 

191 

16,641 

14,804 

1,837 * 

13,223 

11,602 

1,621 * 

Earned $20,000 or more (%) 

37.8 

29.3 

8.5 ** 

32.2 

27.4 

4.8 

40.5 

31.1 

Q 4 *** 

28.0 

22.5 

5.5 * 

Quarter 10 

Ever employed (%) 

70.4 

64.3 

6.0 * 

60.4 

63.3 

-2.9 

64.8 

61.7 

3.1 

68.9 

64.0 

4.9 

Earnings ($) 

5,385 

4,096 

1,289 *** 

3,966 

4,105 

-139 

3,857 

3,470 

387 

3,318 

3,142 

176 

Sample size (total = 2,564) 

349 

341 


242 

237 


353 

344 


349 

349 



SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York State Department of Labor for Per Scholas and 
St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma Employment 
Security Commission for Madison Strategies Group sample members. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 



the time participants spent in occupational skills training, these measures may not fully repre- 
sent the effects of the WorkAdvance programs. The remaining sets of measures break the fol- 
low-up period up into pieces — Year 1 (Quarters 2 to 5), 16 Year 2 (Quarters 6 to 9), and the first 
quarter of Year 3 (Quarter 10) — to help isolate any impacts observed in the post-occupational 
skills training period. 17 Figure 5.1 and Figure 5.2 present impacts on employment and earnings, 
respectively, in unemployment insurance-covered jobs by quarter relative to random assigmnent 
for each site. 

• Consistent with the findings from the Year 2 Survey, the WorkAdvance 
program at Per Scholas produced impacts on employment in and earn- 
ings from unemployment insurance-covered jobs. The impact on earn- 
ings grew stronger over time. 

In the quarter of random assigmnent and the quarter following random assigmnent 
(Quarter 2), more control group members at Per Scholas were employed than WorkAdvance 
group members (see Figure 5.1). This was expected, given that many WorkAdvance group 
members were enrolled in occupational skills training in those quarters. By Quarter 3, though, 
WorkAdvance group members were more likely to be employed (although the difference is not 
statistically significant). Looking at Year 1 overall, 77 percent of Per Scholas WorkAdvance 
group members were ever employed, compared with 7 1 percent of control group members, a 
statistically significant increase of 6 percentage points (Table 5.2). This effect on employment 
was similar in Year 2 and the first quarter of Year 3. 

In the full follow-up period, WorkAdvance group members at Per Scholas earned 
$32,470 compared with average earnings of $27,284 for control group members, a statistically 
significant increase of $5,186 (Table 5.2). Most of this increase in earnings came later in the 
follow-up period. Figure 5.2 shows that at Per Scholas, WorkAdvance group members did not 
begin earning more than control group members until Quarter 4 — slightly later than the trend 
seen for employment. After Quarter 4, earnings increased for the most part for both research 
groups, but WorkAdvance group members earned more in all four quarters of Year 2 and in the 
first quarter of Year 3. In the first quarter of Year 3, WorkAdvance group members earned 
$1,289, or 31 percent, more than the control group. Combining this with the trend seen on em- 
ployment, it appears Per Scholas WorkAdvance group members had higher earnings than con- 
trol group members because they had higher wages (as the survey suggests), and not just be- 
cause more of them were working. 

16 Quarter 1 is the quarter in which sample members were randomly assigned. Depending on when in the 
quarter a sample member was randomly assigned, some employment and earnings in that quarter may precede 
the sample member’s entry into the study. For that reason, Year 1 measures do not include Quarter 1. 

17 An alternative way to break up the follow-up period would be into Year 1 (Quarters 2 to 5) and the post- 
vocational training period (Quarters 6 to 10). The Quarters 6 to 10 measures are not shown in the report, but in 
general, they follow a similar pattern to that seen in the Y ear 2 impact estimates. 
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Figure 5.1 (continued) 

SOURCES: MDRC calculations from unemployment insurance administrative records provided by 
New York State Department of Labor for Per Scholas and St. Nicks Alliance sample members; 
Ohio Department of Job and Family Services for Towards Employment sample members; and 
Oklahoma Employment Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Sample sizes may vary because of missing values. 


• WorkAdvance group members at St. Nicks Alliance were more likely 
than control group members to have worked in an unemployment in- 
surance-covered job in Year 1, but the effect faded by Year 2. No statis- 
tically significant effects were measured on earnings. 

In Year 1, WorkAdvance group members at St. Nicks Alliance were more likely to be 
working than control group members — 78 percent of WorkAdvance group members com- 
pared with 70 percent of control group members were employed. A negative impact is seen on 
employment stability during that time period. Only 2 1 percent of WorkAdvance group mem- 
bers were employed in all four quarters of Year 1 compared with 28 percent of control group 
members. This may relate to the project-based nature of environmental remediation jobs. Work- 
ers in the sector often hold shorter-term jobs and have to seek employment on a new project or 
with a new employer after one job ends. By Year 2, the estimated effects on both employment 
and employment stability at St. Nicks Alliance had faded to statistical insignificance (Table 
5.2). 


There is no statistically significant difference on earnings over the full follow-up period 
at St. Nicks Alliance. WorkAdvance group members earned an average of $27,782, $201 less 
than control group members (Table 5.2). Figure 5.2 shows that the two research groups had 
similar earnings throughout most of the follow-up period. 

• At Madison Strategies Group (one of the providers that initially offered 

the placement-first track), the WorkAdvance program produced im- 

pacts on earnings in Year 2. The program had little effect on employ- 
ment, perhaps because the control group had the highest rate of unem- 
ployment insurance-covered employment in the full follow-up period 
across the sites. 
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Mean earnings ($) Mean earnings ($) 


The WorkAdvance Study 
Figure 5.2 

Quarters 1 to 10 Impacts on Mean Earnings from an Unemployment Insurance-Covered Job, by Relative Quarter and Site 


Per Scholas 



Quarter relative to random assignment 



Quarter relative to random assignment 



Quarter relative to random assignment 


Towards Employment 



Quarter relative to random assignment 


(continued) 



Figure 5.2 (continued) 

SOURCES: MDRC calculations from unemployment insurance administrative records provided by 
New York State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio 
Department of Job and Family Services for Towards Employment sample members; and Oklahoma 
Employment Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Sample sizes may vary because of missing values. 


The WorkAdvance program at Madison Strategies Group did not produce an impact on 
employment in any of the follow-up periods shown in Table 5.2. Figure 5.1 shows that the per- 
centage of WorkAdvance group members employed peaked in Quarter 4 and then began to de- 
crease through Quarter 10. The employment rate for control group members remained fairly 
consistent throughout the follow-up period, and in most quarters, fewer (although the differ- 
ences are not statistically significant) control group members were working than WorkAdvance 
group members. 18 These high control group employment rates are probably a reflection of the 
strong economic conditions in Tulsa during the study period (see Appendix Figure B. 1). 

Although there was no effect on employment, the story on earnings is more positive. 
The WorkAdvance program at Madison Strategies Group increased earnings by $2,552 over the 
control group average during the full follow-up period, although this increase is not statistically 
significant. WorkAdvance group members earned an average of $33,759 in this time period, 
compared with earnings of $31,207 for control group members. When looking just at Year 2 
(the period after occupational skills training), however, WorkAdvance produced statistically 
significant impacts on earnings and earnings above $20,000. WorkAdvance group members 
earned an average of $16,641 in Year 2, an increase of $1,837 (or 12 percent) over the control 
group average. An impact of this size is on par with impacts measured in older job training 
evaluations, such as the Job Training Partnership Act, but at this point in the follow-up period is 
far weaker than the impacts measured in the Sectoral Employment Impact Study. 19 Figure 5.2 
shows that WorkAdvance group members earned significantly more than control group mem- 
bers in Quarters 8 and 9. 


Appendix Figure F.4 shows impacts on unemployment insurance-covered employment in the targeted 
sectors — manufacturing and transportation — at Madison Strategies Group. Beginning in the quarter of ran- 
dom assignment, employment in the targeted sector increased for both research groups, but WorkAdvance 
group members were significantly more likely to be employed in the targeted sector than control group mem- 
bers throughout the follow-up period. 

19 Bloom et al. (1997); Maguire et al. (2010). 
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• The WorkAdvance program at Towards Employment (the other pro- 
vider that initially offered the placement-first track) produced an impact 
on employment in Year 1. This effect on employment grew stronger in 
Year 2, when the provider also produced impacts on earnings. 

At Towards Employment, WorkAdvance produced an increase in the percentage of 
WorkAdvance group members who worked in Quarters 2 to 10 compared with the control 
group: 93 percent of WorkAdvance group members were employed compared with 87 percent 
of control group members (see Table 5.2). 20 Figure 5.1 shows that the employment rate was 
fairly consistent across research groups through Quarter 6. After that, the employment rate for 
the control group dropped somewhat, while the employment rate rose for the WorkAdvance 
group. WorkAdvance group members worked at a significantly higher rate than control group 
members in Quarters 8 and 9. 

The effects on earnings follow a similar pattern. Figure 5.2 shows that in Quarters 7 and 
8, Towards Employment WorkAdvance group members earned significantly more than control 
group members. 21 In Year 2 overall, Towards Employment produced an impact on earnings and 
high earnings (Table 5.2). Around 28 percent of WorkAdvance group members earned $20,000 
or more in Year 2, a statistically significant increase of 6 percentage points over the control 
group. 


Overall Assessment 

This section shows that the impacts on employment and earnings varied across the sites. 
Whenever there are mixed findings across multiple outcomes it can be difficult to assess the 
overall results. The results at this stage for Per Scholas and St. Nicks Alliance are clear — the 
former produced large impacts on employment, wages, and earnings, while the latter produced 
no impacts on employment or earnings. However, the results at Madison Strategies Group and 
Towards Employment are mixed, with evidence of stronger impacts emerging in Year 2. 

One very conservative approach to assessing the overall results is to use what are 
known as “confirmatory outcomes.” Confirmatory analyses involve the estimation of impacts 
on a limited set of prespecified outcomes, which are subjected to multiple comparisons adjust- 


“°Appendix Figure F.4 shows that throughout the follow-up period, the impacts on employment at To- 
wards Employment for the manufacturing sector are larger and more consistent than the impacts for the health 
care sector. WorkAdvance increased employment in the manufacturing sector (among sample members ran- 
domly assigned into the manufacturing sector program) from Quarter 3 to Quarter 10. 

“'Appendix Table F.4 shows impacts on employment and earnings by random assignment sector at To- 
wards Employment. Among sample members randomly assigned in the manufacturing sector, WorkAdvance 
increased employment in an unemployment insurance-covered job in Year 2. In the first quarter of Year 3, the 
estimated impact on employment among sample members randomly assigned in the health care sector is statis- 
tically significant. 
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ments. 22 This type of analysis measures the overall success of a program at a very high standard 
of evidence. Table 5.3 shows the impacts on the confinnatory outcomes prespecified in Work- 
Advance, and Box 5.1 provides further explanation of the analysis and the results. From this 
perspective, and at this interim stage, only Per Scholas passed the confirmatory impacts test. 
Still, despite not producing impacts on the confinnatory measures, the evidence above shows 
that Madison Strategies Group and Towards Employment did increase employment and earn- 
ings, particularly late in the follow-up period. The rest of this chapter will show that these two 
sites also produced impacts on measures of other job characteristics and advancement. 


Impacts on Wage Growth and Advancement 

One of the key innovations of the WorkAdvance model was the addition of advancement- 
focused postemployment services to sectoral training. Many low-wage workers report that their 
jobs offer few opportunities for advancement and will not help them meet their career goals. 23 
To combat this, the postemployment services were designed to provide coaching to help em- 
ployed participants retain and advance in their jobs. 24 Table 5.4 presents impacts on financial 
and nonfmancial advancement at sample members’ current or most recent jobs. The WorkAd- 
vance evaluation as a whole was set up to test the full array of WorkAdvance services, and not 
the effectiveness of postemployment services or advancement coaching on their own. There- 
fore, it is hard to say with confidence that any effects are due specifically to the postemploy- 
ment services. 

The primary measure of advancement in Table 5.4 is the change in sample members’ 
hourly wages since study entry. This measure compares survey respondents’ hourly wages at 
the time of the survey interview to their most recent hourly wages reported at the time they en- 
tered the study. Changes in hourly wages were measured only for respondents who had worked 
within the two years preceding study entry and who were currently employed at the time of the 
survey interview. 

The other advancement measures presented in Table 5.4 are based on nonfmancial job 
characteristics and are related to changes in job title and increases in job responsibilities and 
skills. As discussed earlier, it was hypothesized that little, if any, advancement in terms of wage 
growth would occur in the short follow-up period covered by the Year 2 Survey. The nonfinan- 
cial advancement measures, however, represent early indicators of future wage growth or signs 
of higher-quality employers and jobs. One of the main potential benefits of sectoral programs, 
such as WorkAdvance, is their connection with higher-quality employers. Participants who are 


22 See Olken (2015) for a discussion. 

23 Tompson, Benz, Agiesta, and Junius (2013). 

This coaching was supposed to occur throughout the entire WorkAdvance program. 
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The WorkAdvance Study 
Table 5.3 


Impacts on Confirmatory Outcomes, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Among respondents to the Year 2 Survey 

Currently employed (%) 74.4 

64.3 

|Q | *** 

65.2 

65.6 

-0.5 

77.5 

72.1 

5.4 

64.4 

58.3 

6.1 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 


Among full research sample 

Earnings in first quarter of Year 3 ($) 

5,416 

4,064 

1,351 *** 

3,938 

4,133 

-195 

3,863 

3,463 

400 

3,314 

3,146 

167 

Sample size (total = 2,564) 

349 

341 


242 

237 


353 

344 


349 

349 



SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Gray shading indicates that the statistical test 
withstood the Westfall-Young adjustment for multiple comparisons. After the adjustment, the impact on current employment at Per Scholas is significant at the 
10 percent level and the impact on earnings in the first quarter of Year 3 is significant at the 1 percent level. 

Sample sizes may vary because of missing values. 




Box 5.1 

Confirmatory Economic Outcomes 


In recent years, the issue of multiple test bias has become more prominent in both the academic lit- 
erature and the field of program evaluation more generally. Every time one estimates an impact on 
an outcome, there is a precisely defined probability (conventionally 10 percent in such studies as 
WorkAdvance) of concluding that a program has had an impact when the observed difference is 
simply due to chance. Since researchers typically examine many outcomes, the probability that at 
least one estimate will be statistically significant, simply by chance, can get very high. 

One approach to this problem is to conduct fewer impact estimates and to state in advance which 
tests will be conducted." The WorkAdvance research team followed this approach, prespecifying 
two measures in the analysis planning phase — current employment at the time of the Y ear 2 Sur- 
vey and earnings in an unemployment insurance-covered job using the last quarter of available data 
— as the most likely to be affected if the program were successful. 1 Impact estimates on these 
measures were also subjected to Westfall-Young multiple comparisons correction. (See Appendix 
A for more details.) 

Statistically significant impacts on these “confirmatory” measures represent the highest level of ev- 
idence of the success of the programs with the available amount of follow-up data. If the program 
does not pass the confirmatory test but produces statistically significant impacts on other measures, 
it does not mean that the program was unsuccessful or that it will not lead to increases in employ- 
ment or earnings in the future. It simply means that the program passed a lower standard of evi- 
dence given the available amount of follow-up data. 

Table 5.3 shows that at the Per Scholas site, 74 percent of WorkAdvance group respondents were 
currently employed at the time of the Y ear 2 Survey compared with 64 percent of control group re- 
spondents, a statistically significant increase of 10 percentage points. Further, WorkAdvance in- 
creased earnings in an unemployment-insurance covered job in the first quarter of Year 3 by 33 
percent over the control group average.* WorkAdvance group members earned an average of 
$5,416 in the first quarter of Year 3, compared with average earnings of $4,064 for control group 
members. Both of these impacts withstood the Westfall-Young adjustment. 5 

At Madison Strategies Group and Towards Employment, the WorkAdvance programs did increase 
both employment and earnings, although not by a statistically significant amount. Negative differ- 
ences are seen on both confirmatory outcomes at St. Nicks Alliance. 

*01ken (2015). 

These measures were selected because employment and earnings are the key outcomes in the WorkAdvance 
logic model and the selected points allow for the most time to pass after sample members engaged in occupational 
skills training. Initially, the earnings measure chosen was for Quarter 7. However, as more follow-up data became 
available, it was decided to stick with the concept of using the latest quarter of UI wage data as the confirmatory 
measure. 

*The results in Table 5.3 differ slightly from the results presented elsewhere in this chapter. In order to perform 
the Westfall- Young correction, a different set of common, cross-site co variates was used to obtain the impact esti- 
mates in Table 5.3. The other tables in this chapter used site-specific covariates to estimate the site-specific impacts. 

Alter the adjustment, the impact on current employment is significant at the 10 percent level and the impact 
on earnings in the first quarter of Year 3 is significant at the 1 percent level. 
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The WorkAdvance Study 
Table 5.4 

Year 2 Impacts on Wage Growth and Advancement in Current or Most Recent Job, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 




Difference 



Difference 



Difference 



Difference 

Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Change in hourlv wage since RA 

Did not have a job before RA and/or 



tt 









t 

not employed at survey interview 3 

Flad a job before RA and employed 

37.9 

44.9 

-7.0 

47.5 

43.0 

4.5 

28.2 

31.9 

-3.7 

46.8 

49.2 

-2.4 

at survey interview and: 

Flourly wage increased by: 













$8.00 or more 

16.4 

8.4 

8.0 ** 

8.1 

8.6 

-0.5 

10.0 

6.3 

3.7 

1.5 

2.5 

-1.0 

$5.00-$7.99 

11.3 

8.1 

3.2 

7.1 

3.7 

3.3 

11.3 

9.2 

2.1 

4.3 

5.7 

-1.4 

$2.00-$4.99 

Flourly wage within $2.00 of 

8.9 

11.3 

-2.4 

13.0 

11.2 

1.9 

20.9 

17.4 

3.5 

16.1 

7.8 

8.3 ** 

baseline wage 

15.9 

15.0 

0.9 

12.3 

17.0 

-4.7 

22.7 

25.2 

-2.5 

24.9 

25.3 

-0.3 

Flourly wage decreased 

9.7 

12.4 

-2.7 

12.1 

16.6 

-4.5 

6.8 

10.0 

-3.2 

6.3 

9.6 

-3.2 

Job skills at current or most recent iob 

Scope of work has increased 

50.0 

39.7 

10.3 ** 

37.6 

40.4 

-2.8 

47.0 

49.2 

-2.3 

42.7 

33.7 

9 0** 

Job title changed since job started 

Offered many opportunities for career 

19.4 

15.2 

4.2 

15.0 

14.8 

0.2 

19.6 

25.3 

-5.8 

15.0 

16.2 

-1.2 

advancement 

66.1 

53.1 

13.0 *** 

61.1 

59.3 

1.8 

66.1 

55.4 

10 2 *** 

55.6 

47.2 

g 4 ** 

Obtained new skills while working job 

72.3 

61.2 

\ \ 2 *** 

65.0 

56.0 

9.0 * 

67.4 

59.4 

8.0 ** 

56.2 

56.0 

0.2 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 
for categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

a Sample members who did not work within the two years preceding RA or who were not employed at the time of the survey interview are included in this 
category. 



placed in jobs — even if the jobs are entry level or initially low wage — with employers who 
offer advancement opportunities have the potential for earnings gains in the future. It should be 
noted that any impacts seen on these measures may have occurred either because the jobs ob- 
tained by WorkAdvance group members did offer more opportunities for advancement or be- 
cause WorkAdvance group members were more aware of and likely to seek out advancement 
opportunities than control group members since they were trained by WorkAdvance providers 
to do so. Either way, the impacts are noteworthy. 

• WorkAdvance at Per Scholas resulted in more WorkAdvance group 
members than control group members having increased their wages and 
holding jobs with opportunities for advancement. 

Over one-third of Per Scholas WorkAdvance group members reported having a higher 
wage at the time of the survey interview than the most recent wage they had at study entry. Spe- 
cifically, the provider produced an impact of 8 percentage points on wage increases of $8 or 
more per hour over the control group average — 16 percent of WorkAdvance group members 
reported a wage increase at or above this level. Some of this may be due to WorkAdvance 
group members breaking into the IT sector and the associated high wages of jobs in that sector 
(evidence presented in Chapter 6 indicates that IT sector jobs paid more than non-IT sector jobs 
for the employed Per Scholas WorkAdvance respondent sample). Before entering the study, 
only 8 percent of sample members at Per Scholas had previous employment in the IT sector. 
The sector employment rate increased dramatically for the WorkAdvance group during the fol- 
low-up period — 61 percent of WorkAdvance group members were employed in the IT sector 
at their current or most recent job, compared with 19 percent of control group members (as 
shown in Table 5.2). 

Additionally, WorkAdvance at Per Scholas had statistically significant impacts on sev- 
eral measures of nonfinancial advancement. WorkAdvance group members were 10 percentage 
points more likely to work a job in which their scope of work had increased and 1 1 percentage 
points more likely to have obtained new skills while working compared with control group 
members. Further, 66 percent of WorkAdvance group members believed their job offered many 
opportunities for career advancement, an increase of 13 percentage points over the control 
group average. 

• The WorkAdvance programs at St, Nicks Alliance, Madison Strategies 
Group, and Towards Employment had little to no impact on wage 
growth relative to the control groups. However, the WorkAdvance pro- 
grams at all three sites produced impacts on at least one of the nonfi- 
nancial measures of advancement. 
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At Towards Employment, WorkAdvance group members were significantly more like- 
ly to have had a wage increase of between $2.00 and $4.99 per hour since study entry compared 
with control group members. The WorkAdvance program at Towards Employment also pro- 
duced impacts of 9 and 8 percentage points, respectively, on the likelihood that WorkAdvance 
group members agreed they worked at jobs where their scope of work increased and at jobs that 
offered opportunities for career advancement. 

WorkAdvance group members at St. Nicks Alliance and Madison Strategies Group 
were more likely to report that they obtained new skills while working at their current or most 
recent job, with increases of 9 and 8 percentage points, respectively, over the control group av- 
erages. In addition, Madison Strategies Group WorkAdvance group members were 1 1 percent- 
age points more likely to report that their jobs offered many opportunities for career advance- 
ment compared with the control group. These nonfmancial advancement measures are charac- 
teristics of higher-quality jobs. 


Impacts on Job Benefits and Work Schedule 

Although wages are often thought of as the primary indicator of high-quality jobs, other job 
characteristics, including employer-offered benefits and standard or stable work schedules, are 
also important determinants of job quality. Low- wage workers are offered employer-provided 
benefits at a lower rate than workers with higher wages. These benefits include paid leave, re- 
tirement plans, and health insurance coverage, among others. 25 Further, the number of “under- 
employed” workers — individuals who work part time but are available to work full time — 
increased substantially during and following the Great Recession. 26 Recent reports and legisla- 
tion have highlighted the challenge of irregular job schedules and shifts faced by these under- 
employed, and often low- wage, workers. 27 These types of job characteristics are part of what 
determines the quality of a job, as well as affecting workers’ assessments of job satisfaction. 
Table 5.5 presents impacts on job characteristics of survey respondents’ current or most recent 
jobs. 


• At Madison Strategies Group, WorkAdvance group members reported 
more employer-offered benefits, higher levels of overall job compensa- 
tion, and higher levels of job satisfaction than control group members. 


25 Bureau of Labor Statistics (2014). 

26 Mayer (2014). 

27 See Golden (2015). The Schedules That Work Act was introduced in Congress in 2014 but remains in 
committee (see U.S. Congress, 2015). 
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Table 5.5 


Year 2 Impacts on Current or Most Recent Job Characteristics, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Very or somewhat satisfied with current 

or most recent job 

75.6 

63.7 

H 9 *** 

68.8 

67.6 

1.2 

77.7 

69.2 

8.6 ** 

60.8 

65.6 

-4.8 

Employer-provided benefits 

Sick days with full pay 

40.7 

36.9 

3.8 

43.6 

42.2 

1.4 

46.8 

42.0 

4.8 

35.0 

32.4 

2.7 

Paid vacation 

37.6 

34.9 

2.8 

40.2 

41.5 

- 1.2 

60.3 

51.6 

g 7 ** 

41.1 

39.4 

1.8 

Paid holidays 

41.5 

36.8 

4.7 

46.5 

42.9 

3.6 

60.7 

53.7 

7.0 * 

48.3 

41.9 

6.4 

Dental benefits 

38.5 

35.9 

2.7 

32.1 

33.4 

-1.3 

57.4 

50.9 

6.5 

40.0 

36.6 

3.5 

Retirement plan 

36.1 

28.9 

7.3 * 

29.8 

33.2 

-3.5 

48.1 

41.5 

6.6 

39.3 

35.8 

3.5 

Health plan or medical insurance 

42.5 

39.8 

2.7 

37.8 

40.9 

-3.1 

65.2 

56.4 

8.8 ** 

43.0 

40.9 

2.1 

Enrolled 

70.2 

59.6 

10.6 

60.2 

56.0 

4.3 

67.8 

67.9 

-0.2 

62.6 

61.2 

1.4 

Tuition reimbursement 

20.4 

18.5 

2.0 

14.4 

17.7 

-3.3 

29.0 

21.1 

79 ** 

22.2 

22.0 

0.2 

Hourly wage above $12 and 

employer-provided health insurance 

33.4 

27.9 

5.5 

27.2 

26.6 

0.6 

40.3 

30.4 

99 ** 

20.7 

16.7 

4.0 

Work schedule 

Regular shiff* 

73.6 

63.6 

10.0 ** 

71.0 

63.5 

7.5 

76.9 

78.4 

-1.5 

71.7 

64.0 

7.7 * 

Rotating, split, or irregular shift 

19.8 

25.0 

-5.2 

18.5 

24.8 

-6.3 

15.9 

15.2 

0.7 

16.5 

20.9 

-4.5 

Job type b 

Work for "temp" agency 

14.9 

9.2 

5.7 ** 

8.0 

9.0 

- 1.0 

12.5 

18.0 

-5.5 * 

12.8 

11.8 

1.0 

Work for staffing agency 

19.7 

8.8 

10 9 *** 

13.4 

9.1 

4.3 

12.8 

18.7 

- 6.0 * 

14.5 

10.7 

3.8 

Occasional, odd job 

14.0 

15.2 

- 1.2 

12.5 

15.6 

-3.1 

6.7 

9.6 

-3.0 

10.3 

15.6 

-5.3 * 

Work for a friend or family member 

5.2 

5.4 

- 0.2 

5.5 

6.0 

- 0.6 

8.5 

6.5 

2.0 

6.6 

6.4 

0.2 

Regular permanent job 

71.4 

64.1 

7.3 * 

70.8 

66.3 

4.5 

86.3 

82.0 

4.4 

75.6 

68.1 

7.5 ** 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



(continued) 





Table 5.5 (continued) 

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Sample sizes may vary because of missing values. 

Italics indicate the metric is not among the full sample shown in the table. Therefore, the 
measure is nonexperimental and statistical tests were not performed. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

a A regular shift includes those worked in the daytime, evening, or nighttime. 
b Respondents could select more than one job type, so percentages may sum to more than 100 
percent. 


WorkAdvance group members at Madison Strategies Group reported significantly more 
employer-offered benefits than the control group, including increases of between 7 and 9 per- 
centage points on the likelihood that respondents’ employers offered paid vacation, paid holi- 
days, health insurance, 28 and tuition reimbursement. Madison Strategies Group directly targeted 
employers and jobs that offered benefits. The provider was focused on increasing total compen- 
sation, rather than just hourly wages, and sometimes placed participants in jobs with slightly 
lower hourly wages if the job offered benefits. This is seen in the impact of almost 10 percent- 
age points on the likelihood that respondents had a job with an hourly wage above $12 per hour 
and were offered health insurance by their employer. 

WorkAdvance group members at Madison Strategies Group also reported significantly 
higher levels of job satisfaction than control group members. Around 78 percent of WorkAd- 
vance group members reported being very or somewhat satisfied with their current or most re- 
cent job, an increase of 9 percentage points over the control group average. 

• Per Scholas WorkAdvance group members reported higher levels of job 
satisfaction and more stable work schedules than control group mem- 
bers. The program at Towards Employment produced increases in 
regular permanent employment (versus occasional or odd jobs). 

Per Scholas WorkAdvance group members reported higher levels of job satisfaction (76 
percent) than control group members (64 percent), an impact of nearly 12 percentage points. 

28 Around 65 percent of WorkAdvance group members at Madison Strategies Group reported being of- 
fered health insurance through their employer. This is notable given that less than 30 percent of sample mem- 
bers at that provider reported being covered by health insurance at baseline. In another section of the Year 2 
Survey, respondents were asked whether they were covered by health insurance in the past month and if so, 
what type of health insurance they were covered by (see Appendix Table F.3). Fewer respondents in both re- 
search groups at Madison Strategies Group reported being covered by employer-based health insurance in the 
prior month than the rates shown in Table 5.5. The estimated effect is not statistically significant. 
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Additionally, 74 percent of WorkAdvance group members reported having a job with a regular 
shift and 20 percent reported having a job with a rotating, split, or irregular shift (compared with 
64 percent and 25 percent of control group members, respectively). The estimated difference on 
working a regular shift is statistically significant. As mentioned, the issue of low-wage workers 
having jobs with irregular hours and shifts has been a major concern for policymakers late- 
ly, making this impact especially noteworthy. 

The Per Scholas program had little effect on employer-provided benefits, with the ex- 
ception of retirement plans. This may be related to the fact that entry-level jobs in the IT indus- 
try are often offered through “temp” or staffing agencies (the impacts on both of these measures 
are statistically significant). Jobs obtained through these types of agencies do not usually offer 
benefits. 

The Towards Employment program, while not affecting job satisfaction or the receipt 
of employer-offered benefits, did affect individuals’ work schedules and job type: Towards 
Employment WorkAdvance group members were 5 percentage points less likely to report that 
their job is an occasional or odd job and 8 percentage points more likely to report their job is a 
regular permanent job, compared with control group members. 

• The WorkAdvance programs at St. Nicks Alliance produced no statisti- 
cally significant impacts on job satisfaction, employer-provided benefits, 
and job schedules and types. 

WorkAdvance and control group members at St. Nicks Alliance reported similar levels 
of job satisfaction (69 percent and 68 percent, respectively) and employer-offered benefits. 29 


Impacts on Total Income 

Increases in employment and job earnings do not always translate into higher income levels for 
low- wage workers. 30 As individuals make the transition from unemployment to employment or 
to employment with higher wages, they often lose eligibility for public benefits. 31 In that case, 
job earnings may merely replace previous income from public assistance, rather than increase 


29 Notably, though, St. Nicks Alliance had the highest rate of respondents employed in a unionized job 
across the four sites — 21 percent of WorkAdvance group members and 20 percent of control group members 
(see Appendix Table F.2). While the difference across research groups is not statistically significant, it is still 
important to note that WorkAdvance group members were obtaining unionized jobs, as this is a sign of job 
security. This is probably a reflection of the provider’s focus on the environmental remediation sector. 

30 Total income includes income from job earnings. Supplemental Nutrition Assistance Program (food 
stamps). Social Security Disability Income or Social Security Income, Temporary Assistance for Needy Fami- 
lies (welfare), retirement funds or Social Security, unemployment insurance benefits, child support, and other 
sources. 

3 'Bloom and Michalopoulos (2001). 
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total income. While there can be a societal benefit to a reduction in the use of public assistance 
programs, individuals are not always better off financially. Table 5.6 presents WorkAdvance’s 
impacts on income and sources of income in the month before the survey interview. Survey re- 
spondents were asked to report both their own and their household’s previous month’s income, 
as well as the sources of their household’s income. 32 

• At Per Scholas, WorkAdvance increased the average prior month’s in- 
come by 33 percent over the control group average. WorkAdvance also 
reduced use of several of the most common public assistance programs. 

The average monthly income for Per Scholas WorkAdvance group members was 
$1,580, a large and statistically significant increase of $396 over the control group average of 
$1,185. The bulk of this income appears to stem from job earnings: 80 percent of Per Scholas 
WorkAdvance group members reported having job earnings in the prior month, an increase of 
12 percentage points over the control group average. Further, WorkAdvance at Per Scholas de- 
creased respondents’ use in the prior month of three of the most common public assistance pro- 
grams — Supplemental Nutrition Assistance Program (SNAP) benefits (food stamps), Tempo- 
rary Assistance for Needy Families (TANF), and unemployment insurance benefits 33 — by be- 
tween 4 and 1 1 percentage points. These findings highlight that the WorkAdvance program at 
Per Scholas was able to increase income while also reducing use of public assistance programs, 
a feat only rarely accomplished by employment and training programs. 34 

• The WorkAdvance programs at St. Nicks Alliance, Madison Strategies 
Group, and Towards Employment had little to no effect on total income. 

The income levels for both research groups at Towards Employment are much lower 
than those at the other three sites. WorkAdvance group members at Towards Employment re- 
ported an average income of $1,002 in the month before the survey interview, $17 less than 
control group members, but this effect is not statistically significant. WorkAdvance did increase 


32 The WorkAdvance Year 2 Survey defined a household as the survey respondent and the survey re- 
spondent’s live-in spouse or partner, if applicable. 

^Unemployment insurance benefits data were collected, in addition to the wage data, from the three state 
agencies. Appendix Figure F.5 presents impacts on receipt of unemployment insurance benefits using adminis- 
trative data by quarter relative to random assignment. In the quarter of random assignment, 26 percent of Per 
Scholas WorkAdvance group members and 28 percent of control group members received unemployment 
insurance benefits (the estimated impact is not statistically significant). The percentage of sample members 
receiving unemployment insurance benefits decreased for both groups over time, and in Quarters 6 through 9 
Per Scholas WorkAdvance group members were significantly less likely to receive unemployment insurance 
benefits than control group members. 

34 These findings also suggest, if present trends continue, that a full benefit-cost analysis might find quite 
positive results from the Per Scholas program. 
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The WorkAdvance Study 
Table 5.6 


Year 2 Impacts on Income, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Average total respondent income in 

prior month ($) 

1,580 

1,185 

396 *** 

1,515 

1,425 

90 

1,664 

1,664 

-1 

1,002 

1,019 

-17 

Average total household income in prior 

month 1 * ($) 

2,166 

1,754 

4^2 *** 

1,957 

2,024 

-68 

2,147 

2,270 

-123 

1,406 

1,417 

-11 

Household income source (%) 

Job earnings 

79.7 

67.6 

12.2 *** 

70.1 

73.6 

-3.5 

80.6 

78.9 

1.7 

71.8 

62.5 

9 4 ** 

Food stamps/SNAP 

12.8 

23.8 

-110 *** 

26.3 

29.1 

-2.8 

21.4 

24.8 

-3.5 

43.3 

47.1 

-3.8 

SSI/SSDI 

4.1 

1.6 

2.4 * 

3.8 

6.5 

-2.7 

7.9 

6.1 

1.8 

12.4 

10.5 

1.8 

Welfare/TANF 

4.0 

7.9 

-3.9 * 

8.9 

7.3 

1.7 

1.3 

1.6 

-0.4 

3.7 

6.0 

-2.3 

Housing assistance 

8.9 

11.3 

-2.3 

8.9 

10.3 

-1.4 

5.2 

5.7 

-0.5 

15.4 

13.8 

1.6 

Unemployment insurance 

4.5 

8.1 

-3.6 * 

7.8 

5.2 

2.6 

2.2 

2.6 

-0.4 

4.3 

4.7 

-0.4 

Other 

10.8 

14.9 

-4.1 

14.4 

16.3 

-1.9 

20.1 

21.4 

-1.3 

40.6 

36.5 

4.1 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; SNAP = Supplemental Nutrition Assistance Program; SSI = Supplemental Security 
Income; SSDI = Social Security Disability Income; TANF = Temporary Assistance for Needy Families. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

“Household income includes income from the survey respondent and, if applicable, income from the respondent's live-in spouse or partner. 



the percentage of Towards Employment respondents receiving job earnings in the prior month, 
by 9 percentage points. This impact on income from job earnings differs from Towards Em- 
ployment’s impact on current employment reported earlier in this chapter (about 6 percentage 
points, but the effect is not statistically significant). This difference may be because the income 
sources shown in Table 5.6, including job earnings, cover both the respondent and any live-in 
spouse or partner, while the current employment measure reflects only earnings from the sam- 
ple member. 

Neither the WorkAdvance program at St. Nicks Alliance nor the WorkAdvance pro- 
gram at Madison Strategies Group had a statistically significant effect on income levels. In addi- 
tion, neither program reduced WorkAdvance group members’ use of public assistance pro- 
grams. 35 


Impacts on Life Satisfaction and Material Hardship 

Many low-wage workers report feeling pessimistic about their financial situation and worry 
about being able to meet daily living expenses. 36 These living expenses include rent or mortgage 
payments, bill payments, and medical expenses, among others. Table 5.7 presents impacts on 
the secondary economic outcomes of life satisfaction and material hardship. These outcomes are 
considered secondary because they are thought to be affected only after individuals first find 
jobs and increase their earnings. 

• WorkAdvance group members at Per Scholas reported higher levels of 
life satisfaction and of having an improved financial situation than con- 
trol group members. WorkAdvance also decreased the number of mate- 
rial hardships respondents at Per Scholas faced in the past year. 

Approximately 65 percent of Per Scholas WorkAdvance group members reported being satis- 
fied with their lives compared with 44 percent of control group members, a statistically signifi- 
cant impact of 21 percentage points. 37 Per Scholas WorkAdvance group members were also less 
likely than control group members to have reported facing more than five material hardships in 


35 Appendix Figure F.5 (using administrative records data) shows that at St. Nicks Alliance and Madison 
Strategies Group, the two research groups had similar levels of unemployment insurance benefits receipt. In 

Quarter 2, however, WorkAdvance group members at St. Nicks Alliance were more likely than control group 
members to receive unemployment insurance benefits. It is common that workers in the environmental remedi- 
ation field apply for unemployment insurance benefits when they are between projects. 

36 Tompson, Benz, Agiesta, and Junius (2013). 

37 Survey respondents were asked on a scale from 1 to 10 — with 1 being completely dissatisfied and 10 
being completed satisfied — how satisfied they were with their lives. Responses of 7 to 10 were counted as 

“satisfied.” 
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The WorkAdvance Study 
Table 5.7 


Year 2 Impacts on Life Satisfaction and Material Hardship, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Life satisfaction ' 1 

Dissatisfied 

6.6 

13.9 

ttf 

j~j 4 *** 

10.1 

11.7 

-1.5 

11.0 

12.1 

- 1.0 

13.8 

14.8 

- 1.0 

Neither dissatisfied nor satisfied 

28.3 

42.3 

-14.0 *** 

32.6 

39.8 

-7.2 

31.4 

29.7 

1.7 

37.2 

38.3 

- 1.1 

Satisfied 

65.1 

43.8 

2 i 3 *** 

57.3 

48.5 

8.8 

57.6 

58.3 

-0.7 

49.0 

46.9 

2.2 

Improved financial situation 

76.0 

61.6 

14 4 *** 

70.7 

63.4 

7.3 

73.8 

69.8 

4.0 

61.3 

58.4 

2.9 

Number of material hardships in the past 
12 months b 

0 

51.7 

47.6 

tt 

4.2 

46.8 

43.9 

2.9 

39.2 

38.4 

0.8 

38.5 

33.0 

f 

5.5 

1-2 

39.7 

37.4 

2.3 

33.5 

35.9 

-2.4 

35.6 

36.4 

- 0.8 

35.3 

42.5 

-7.2 

3-4 

7.8 

10.1 

-2.3 

14.7 

15.2 

-0.5 

16.2 

16.2 

0.1 

16.5 

19.2 

-2.7 

5 or more 

0.8 

4.9 

.4 i ** 

4.9 

5.0 

- 0.1 

9.0 

9.0 

0.0 

9.6 

5.3 

4.4 

Worry often about being able to meet 

normal monthly living expenses 

31.1 

40.8 

_g ^ ** 

48.1 

46.2 

1.9 

32.6 

37.5 

-4.8 

49.2 

49.2 

0.0 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Sample sizes may vary because of missing values. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall- Young adjusted p-values were used 
for categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

“Respondents were asked to report their life satisfaction on a scale from 1 to 10, with 1 being completely dissatisfied and 10 being completely satisfied. 
Respondents who answered 1-3 were counted as dissatisfied; respondents who answered 4-6 were counted as neither dissatisfied nor satisfied; and respondents 
who answered 7-10 were counted as satisfied. 

b Material hardships include not paying the full amount of rent or mortgage due; being evicted from home or apartment; not paying the full amount due for 
gas, oil, or electricity; having gas, electrical, or oil services turned off; having phone services disconnected because payments were not made; moving in with 
family and friends; and being unable to afford a visit to a doctor or dentist. 



the past year. These large impacts on secondary economic measures reaffinn the overall effec- 
tiveness of the WorkAdvance program at Per Scholas. 

• Few to no significant effects were seen on life satisfaction and material 
hardship for the other three WorkAdvance programs. 

At St. Nicks Alliance and Madison Strategies Group, the WorkAdvance programs pro- 
duced no statistically significant effects on life satisfaction or material hardship. WorkAdvance 
group members at both sites reported having an improved financial situation compared with the 
control group, although neither provider produced a statistically significant impact on that 
measure. 

The effects of the WorkAdvance program at Towards Employment on material hard- 
ship are mixed. The number of material hardships respondents faced in the past 12 months dif- 
fered significantly across research groups. More WorkAdvance group members reported having 
no (39 percent) and five or more (10 percent) material hardships in the past year than control 
group members (33 percent and 5 percent, respectively). 

Conclusion 

This chapter shows that the impacts on employment and earnings produced by the WorkAd- 
vance programs varied across the sites. The story at two sites is fairly straightforward. The Per 
Scholas WorkAdvance program was successful at increasing both employment and earnings 
(including both confirmatory outcomes), as well as producing impacts on several secondary 
economic outcomes. St. Nicks Alliance had little effect on economic outcomes throughout the 
follow-up period. The story at Towards Employment and Madison Strategies Group (the two 
sites that initially operated the “placement-first track”) is somewhat more complicated. There is 
some indication that the WorkAdvance programs at these two sites increased employment, 
earnings, or both for at least part of the follow-up period. Madison Strategies Group increased 
wages above the $12 and $15 per hour thresholds, and both providers also produced improve- 
ments in job characteristics. More follow-up data is needed to clarify the longer-tenn pattern of 
effects at these two sites. 

These findings lead to a few questions. How was Per Scholas able to produce impacts 
on employment and earnings? Why were the other three providers not able to produce as large 
or as consistent impacts? What other factors may have led to this variation in impacts? What 
was the role of the sector in explaining the variation in impacts across the sites? Did the impacts 
grow as the newer providers improved their service offerings and as the two “mixed” sites 
moved to a predominantly training-first model? Did the impacts vary across the sites because 
providers targeted different populations? The next chapter will try to answer these questions by 
examining some possible explanations for what led to the variation in impacts by sector, site, 
time, and subgroup. 
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Chapter 6 

Exploring Sources of Variation in the Economic Impacts 


Introduction 

Chapter 5 described the impacts of WorkAdvance on economic outcomes and presented evi- 
dence that strongly suggested that the impacts varied by site: The Per Scholas program pro- 
duced large impacts over several domains; the Madison Strategies Group and Towards Em- 
ployment programs produced much smaller impacts that were less consistent across time and 
outcomes; and St. Nicks Alliance produced few statistically significant impacts on the economic 
outcomes. This chapter highlights several analyses that were conducted to better understand the 
mechanisms that generated these impacts. In other words, this chapter addresses the “how” and 
“why” questions behind the economic impacts. Understanding these mechanisms provides in- 
sights into the causes behind the site-by-site variation. For the WorkAdvance theory of change 
to translate into impacts, providers had to target the right sectors, recruit the right participants, 
and implement the model well. This chapter focuses on three key questions that assess whether 
variation on any of these elements might help explain site variation in economic impacts: 

1 . Were the impacts driven by the targeted sector? That is, were they driven by 
either the proportion of participants placed in targeted sector jobs or the char- 
acteristics of those jobs, or both? 

2. Did the impacts change over time as programs gained more experience and 
as two of the providers moved to a predominantly training- first approach? 

3. Were the impacts affected by the differing characteristics of participants at 
each site? 

While none of the analyses in this chapter provides a definitive explanation for why the 
impacts varied as they did, the chapter gives rise to a number of hypotheses that can be investi- 
gated in future studies. 


Were the Impacts Driven by the Targeted Sector? 

• The weight of the evidence suggests that both the extent and the quality 
of targeted sector placements were important factors in explaining the 
pattern of impacts by site. 

A key element of the theory of change in sector programs is that by concentrating on 
specific sectors with strong labor market demand and good prospects for advancement, program 
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staff members can develop deep relationships with employers, allowing them to place partici- 
pants in better jobs than they could without this specialization. As discussed in Chapter 1, how- 
ever, this mechanism can break down at several points. First, participants must complete the 
training and obtain any requisite certifications. If participants do not complete (or in some cases 
even start) training, it is unlikely that the training- first pathway of the program will achieve its 
goals. 1 The theory of change could also break down if participants complete the training, but 
providers prove unable to place participants in the targeted sector. This could happen because of 
a misreading of the demand in the sector or a provider’s inability to implement the aggressive 
job development — that is, establishing relationships with employers and matching participants 
with positions — that is needed to take advantage of the jobs that do exist. Finally, it is possible 
that all these elements could be in place, but the program could still fail to produce impacts if 
the jobs in the targeted sector are no better than the jobs that individuals would have found in 
other sectors. (Table 6.1 shows the typical job titles in the sectors targeted by the WorkAdvance 
programs.) This section analyzes these dimensions in turn and suggests that several of these fac- 
tors were central to how the varying impacts of WorkAdvance were produced. 

Impacts on Employment in the Targeted Sector 

Figure 6.1 shows the extent to which the WorkAdvance providers increased employ- 
ment in the targeted sector. There is important variation by site, which parallels the overall pat- 
tern of impacts shown in Chapter 5. Per Scholas increased employment in the targeted sector by 
over 40 percentage points. The increase at St. Nicks Alliance is 12 percentage points. Both 
Madison Strategies Group and Towards Employment increased targeted sector employment by 
16 percentage points. 2 

Activity Flow in Targeted Sector 

Figure 6.2 drills deeper by examining the linkages between training and placement rates 
within the targeted sector for the WorkAdvance group. The leftmost boxes within each panel 
depict the expected pathway (indicated with arrows) of WorkAdvance group members. All per- 
centages are for activities within the targeted sector and are out of the full WorkAdvance group 
sample at each site. The focus of this figure is better understanding the various pathways and 
drop-offs between starting services and obtaining a job in the sector. 


'Training completion rates have been low in some past programs. See Gueron and Hamilton (2002). 

2 The targeted sector placement impacts varied over time at Towards Employment. Among the early cohort 
(roughly the first half of enrollees) the impact on employment in the targeted sector was less than half as large 
as it was among later enrollees. The impact on targeted sector employment was more similar between the co- 
horts at the other sites. 
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The WorkAdvance Study 


Table 6.1 

Common Job Titles and Employers for WorkAdvance Respondents Working in the Targeted Sectors 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Sector 

Information technology 

Environmental remediation 
and related fields 

Transportation and manufac- 
turing 

Health care 

Manufacturing 

Common 
job titles 

• Technical support/help 
desk 

• Field technician 

• Computer technician 

• Pesticide technician 

• Exterminator technician 

• Asbestos handler 

• Manhole inspector 

• Truck driver/ 
equipment cleanup 

• Truck driver 

• Computerized numerical 
control (CNC) 
machinist/CNC operator 

• Diesel mechanic 

• Patient care repre- 
sentative 

• Medical billing and 
patient access 
specialist 

• Patient care assistant 

• Medical assistant 

• Phlebotomist 

• Machine operator 

• CNC machinist/CNC 
operator 

• Welder 

Common 

employer 

types 

• Information technolo- 
gy staffing firms 

• Banks 

• Financial firms 

• Cable companies 

• Educational institu- 
tions 

• Pest control firms 

• Environmental 
remediation firms 

• Firms specializing in 
asbestos removal, mold 
removal, and demolition 

• Trucking companies 

• Food and beverage 
manufacturers 

• Welding supply companies 

• Package delivery services 

• Hospitals 

• Nursing homes/ 
assisted living 
facilities 

• Home health care 
agencies 

• Manufacturers of: 
o Metal parts and 
items 

o Automotive parts 
o Plastics 
o Ceramics 


SOURCE: MDRC analysis of responses to the Year 2 Survey. 




The WorkAdvance Study 
Figure 6.1 

Impacts on Employment Rates in Targeted Sector 
at Current or Most Recent Job, by Site 


Per Scholas 


St. Nicks Alliance 


Madison 
Strategies Group 

Towards 

Employment 


0 10 20 30 40 50 

Percentage point change in employment in targeted sector 

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 




The first two panels show Per Scholas and St. Nicks Alliance — the two providers that 
focused on training throughout the entire evaluation period. Nearly 70 percent of WorkAdvance 
group members at Per Scholas reported starting training in the infonnation technology (IT) sec- 
tor. 3 At St. Nicks Alliance, a somewhat lower rate of WorkAdvance group members started 
training in the environmental remediation sector (59 percent). At both providers, there was rela- 
tively little drop-off between starting training in the targeted sector and obtaining a credential in 
the targeted sector (15 percent of respondents at Per Scholas and 1 1 percent of respondents at 


3 As discussed in Chapter 3, the rates of participation reported in the management information system data 
are somewhat higher, suggesting that some of the survey-based estimates are lower due to lack of recall or oth- 
er reporting issues. 
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The WorkAdvance Study 


Figure 6.2 

Activity Flow in Targeted Sector, Among WorkAdvance Group Respondents, 

by Site 


Per Scholas (N = 287) 



targeted sector 8 targeted sector 8 


St. Nicks Alliance (N = 205) 
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targeted sector 8 
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Figure 6.2 (continued) 


Madison Strategies Group (N = 297) 
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sector 3 targeted sector 3 


Towards Employment (N = 286) 


40% 


Started training in targeted sector 


Did not start training in targeted sector 


24% 


Obtained targeted sector credential 


Did not obtain 
targeted sector credential 

I 


sector 


targeted sector 3 


22% WSM 


10% 


18% 

Obtained job in targeted 

Obtained job 

in 

Obtained job in 


targeted sector 3 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: Sample sizes may vary because of missing values. Measures may not sum to 100 percent because of 
missing values. 

Mobs in targeted sector are WorkAdvance group respondents’ current or most recent jobs only. 
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St. Nicks Alliance fell off the expected path at this point). Figure 6.2 reveals, however, that cre- 
dentials were much more likely to translate into targeted sector jobs at Per Scholas. At St. Nicks 
Alliance, 48 percent of WorkAdvance group members started targeted sector training and ob- 
tained a credential in the targeted sector, but only 2 1 percent started training, got a targeted sec- 
tor credential, and also found a job in the sector (compared with 54 percent and 40 percent, re- 
spectively, at Per Scholas). These results suggest that job development and placement strategies 
were much more successful in getting participants into targeted sector jobs at Per Scholas — 
either because jobs were more plentiful in the IT sector or because Per Scholas was better able 
to make use of employer relationships to place individuals once they were credentialed, or a 
combination of the two. One possible reason for the low targeted sector employment rates at St. 
Nicks Alliance is that participants in the environmental remediation training had to wait for 
weeks or even months before receiving their certifications to work in New York. Some individ- 
uals were probably unable to wait so long and may have opted to work in another field. As dis- 
cussed below, another factor may have been the more limited network of employers that St. 
Nick’s Alliance had to work with compared with the other providers. 

The last two panels show Madison Strategies Group and Towards Employment. De- 
spite starting as mixed training-first and placement-first providers, rates of starting training in 
the targeted sector at Madison Strategies Group and Towards Employment were similar to the 
rates at St. Nicks Alliance (but lower than at Per Scholas). However, the rates of those who both 
started training and received a targeted sector credential at those two providers were lower than 
the rates at both Per Scholas and St. Nicks Alliance. Finally, more than three-quarters of those at 
Madison Strategies Group and almost two-thirds of those at Towards Employment who started 
training and obtained a credential found a targeted sector job. (This result is obtained by divid- 
ing the percentage who stayed on the expected pathway until obtaining a targeted sector job by 
the percentage who stayed on the expected pathway only until obtaining a targeted sector cre- 
dential.) Because these providers also featured a placement-first track, a substantial proportion 
of those who never started training in the targeted sector found work in that sector, unlike at Per 
Scholas and St. Nicks Alliance. 4 Overall, the pattern at these providers is that they were able to 
place more individuals into the targeted sector than St. Nicks Alliance, and they used more di- 
verse pathways to do so (sometimes with training and sometimes without). See Box 6. 1 for a 
discussion of the results in Figure 6.2 from another perspective. 


4 Appendix Figure G. 1 shows these same results for the control group. In general, control group members 
at all sites who found targeted sector jobs were less likely to have received a credential than not. 
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Box 6.1 

Analyzing the Results of Figure 6.2 from Another Perspective 

Besides viewing Figure 6.2 as a comparison of how the providers moved participants 
along the pathway, one can look at it from the perspective of how these programs might be 
improved. Figure 6.2 shows that even at Per Scholas, which produced such impressive im- 
pacts, almost one-third of WorkAdvance group respondents reported that they never start- 
ed training in the targeted sector, and almost 40 percent of respondents started the program 
with the expectation of getting an information technology (IT) job but failed to do so.* It is 
important to keep these findings about drop-off in mind when considering how programs 
can be improved in the future. Improvements depend on understanding why some individ- 
uals never went to training and why many who completed training were unable to find a 
job in the targeted sector. No training program is ever going to achieve a perfect place- 
ment rate, but maximizing the percentage of individuals who successfully complete a sec- 
toral program and get a job in the targeted sector is crucial. 


It should be noted that the percentages shown in Figure 6.2 are from responses to the Year 2 Survey. 
The rates of starting training within the targeted sector as reported by the providers’ own program track- 
ing data are much higher (see Table 2.5), but this information was not available from every provider. 
Some of the differences in rates may be due to recall issues in the survey responses. 


Targeted Sector Job Characteristics 

While all the WorkAdvance providers produced impacts on employment in the targeted 
sector, those jobs have to be better than jobs in other sectors to generate impacts on the primary 
economic outcomes. Figure 6.3 compares targeted sector and non-targeted sector jobs on two 
key dimensions of job quality: wages (as measured by the percentage of employed respondents 
earning above $15 per hour — a commonly used threshold for “living wage” jobs) and hours 
(as measured by the percentage working full time). These comparisons are only among Work- 
Advance group members who had worked since random assignment. It should be noted that 
since the goal of the WorkAdvance programs was to place individuals into jobs in the targeted 
sector, it is likely that participants who were the most job-ready at each site were more apt to get 
these jobs. This may explain some of the differences in outcomes between the targeted sector 
jobs and the others. Given this probable “selection bias” into targeted sector jobs, these results 
should be viewed as descriptive rather than representing the causal effects of the sectoral 
placements themselves. To help account for selection bias, outcomes are regression adjusted 


152 




Percentage Percentage 


The WorkAdvance Study 
Figure 6.3 

Comparison of Characteristics of Targeted Sector Jobs and Nonsector Jobs 
Among WorkAdvance Group Survey Respondents Employed Since Random Assignment 


Per Schoias 
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■ Targeted 
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job (N=95) 
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St. Nicks Alliance 
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Madison Strategies Group 



Hourly wage $15.01 or Working full time (%) 
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Towards Employment 


■ Targeted 
sector job 
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□ Nonsector 
job 
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Hourly wage $15.01 or Working full time (%) 
higher (%) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 


NOTES: Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment. Job characteristics refer 
to the current or most recent job. 

Full time is considered working 35 hours or more per week. 

"Nonsector job" is any job that is not in the sector(s) targeted by the WorkAdvance provider. 



based on a range of observable baseline characteristics, but it is likely that unobservable factors 
that are not adjusted for influence these outcomes. 5 

At Per Scholas and St. Nicks Alliance, there is a clear difference in the percentage of 
targeted sector jobs (compared with non-targeted sector jobs) that paid above $15 per hour. At 
Madison Strategies Group (which focused on transportation and manufacturing in Tulsa), there 
is a smaller difference, and at Towards Employment (which targeted health care and manufac- 
turing in northeast Ohio), there is almost no difference in the likelihood of making above $15 
per hour. While $15 per hour is a key policy-relevant threshold, it is important to note that the 
overall cost of living is lower in Tulsa and northeast Ohio than in New York City, and in both 
locations, WorkAdvance group members who worked in targeted sector jobs were substantially 
more likely to work at wages of $12.01 to $15.00 an hour than those working outside the sec- 
tor. Looking simply at the average wage premium (again, comparing targeted sector jobs with 
other jobs obtained by WorkAdvance group members), the targeted sector jobs paid $1.76 per 
hour more at Per Scholas, $3.77 per hour more at St. Nicks Alliance, $2.23 per hour more at the 
Madison Strategies Group, and only $0.86 per hour more at Towards Employment. Thus, from 
this perspective, only at Towards Employment were targeted sector jobs associated with a rela- 
tively small wage premium. 

At all four sites, individuals who obtained targeted sector jobs were more likely to work 
full time at these jobs compared with individuals who obtained non-targeted sector jobs. This 
difference is particularly large at Madison Strategies Group and Towards Employment. 

While the primary outcome of WorkAdvance is earnings, the program sought to im- 
prove other job characteristics as well. Table 6.2 shows that, generally, targeted sector jobs tend 
to have better characteristics across a range of measures compared with non-targeted sector 
jobs. 6 It is important to remember that these comparisons refer to the characteristics of targeted 
sector versus non-targeted sector jobs and do not necessarily indicate the effects of the program. 
Differences in having employer-offered health insurance on the job were especially large at the 
Madison Strategies Group site (where there was also a statistically significant impact on this 
measure, as discussed in Chapter 5). WorkAdvance group members who worked in the targeted 


5 Appendix Table G.2 shows the adjusted and unadjusted differences in job characteristics between target- 
ed sector and non-targeted sector jobs. In addition, as a sensitivity check, the analysis was reproduced using 
propensity score matching. For this analysis, a model was built that predicted having top quartile earnings in 
Quarter 10. This model was fit to the whole WorkAdvance group, which was then stratified by propensity 
score into two groups, above or below the median. More refined stratification would be preferred, but sample 
sizes were limited. The results, shown in Appendix Tables G.3 and G.4, are similar to those shown here, but 
this analysis still does not rule out the possibility that unobservable factors (such as participant motivation) 
might drive the associations described in this section. 

6 Appendix Table G.l makes the same comparison with control group survey respondents. 
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The WorkAdvance Study 
Table 6.2 


Comparison of Characteristics of Targeted Sector Jobs and Nonsector Jobs 
Among WorkAdvance Group Survey Respondents Employed Since Random Assignment 



Per Scholas 


St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

Targeted Nonsector 
Sector Job Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

T argeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Job characteristics 

Offered health plan or medical insurance 
through employer 

50.8 

35.2 

45.1 

39.7 

79.8 

43.1 

58.7 

33.7 

Working regular shift h 

80.6 

73.8 

81.4 

77.8 

84.3 

72.6 

87.9 

69.8 

Very > or somewhat satisfied with job 

85.5 

71.9 

93.5 

66.9 

84.4 

74.1 

69.6 

66.8 

Advancement 

Had a job before RA, employed at time 
of survey interview, and hourly wage 
increased by $8.00 or more 

30.7 

18.6 

32.2 

7.5 

14.2 

13.6 

3.7 

1.5 

Job skills at current or most recent job 

Scope of work increased 

59.7 

42.5 

54.0 

34.7 

56.1 

35.5 

58.6 

31.4 

Offered many opportunities for career 
advancement 

71.9 

67.9 

72.4 

64.4 

77.8 

48.8 

68.0 

55.0 

Obtained new skills while working job 

82.9 

65.5 

84.4 

65.2 

77.2 

56.0 

69.3 

54.6 

Sample size (total = 992) 

111 

95 

66 

117 

189 

96 

143 

109 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment. 

Italics indicate the statistics are not from the full sample. Statistics are among WorkAdvance group respondents who indicated that they had worked for pay 
since random assignment. Job characteristics refer to the current or most recent job. 

Sample sizes may vary because of missing values. 

a "Nonsector job" is any job that is not in the sector(s) targeted by theWorkAdvance provider. 
b A regular shift includes those worked in the daytime, evening, or nighttime. 





sector at the St. Nicks Alliance site reported much higher levels of job satisfaction compared 
with WorkAdvance group members who worked outside the targeted sector. At the Madison 
Strategies Group and Towards Employment sites, targeted sector jobs offered more opportuni- 
ties for advancement. And Per Scholas and St. Nicks Alliance respondents working in the tar- 
geted sector saw particularly large growth from baseline wage levels, which are consistent with 
the wage differences shown in Figure 6.3. 

Overall Assessment 

• The results suggest that the economic impacts of WorkAdvance depend- 
ed critically on increasing targeted sector employment relative to the 
control group. 

WorkAdvance produced larger wage impacts at Per Scholas by placing many more par- 
ticipants into the better-paying IT sector. At the other three sites, targeted sector jobs had 
somewhat higher wages or hours than non-targeted sector jobs. WorkAdvance group members 
who worked in the targeted sector at St. Nicks Alliance earned much higher wages and worked 
more hours than WorkAdvance group members in non-targeted sector jobs, and wages at tar- 
geted sector jobs were also higher than at other jobs at Madison Strategies Group. At Towards 
Employment, 7 there was a smaller wage premium associated with working in the targeted sector 
(at least initially), but respondents who worked in the targeted sector worked more hours. But 
all three providers increased targeted sector employment at a rate much lower than Per Scholas, 
so impacts were insufficient to translate into large economic impacts. It is important to note, 
however, that WorkAdvance group members employed at targeted sector jobs reported that 
these jobs had more potential for advancement compared with non-targeted sector jobs, and it 
may take time for the wage and earnings premiums to emerge, which highlights the need for 
longer-term follow-up data. 

Did the Impacts Change over Time? 

• At the two providers that were running programs in sectors they were 
new to, the impacts strengthened for the late cohort, as the providers 
gained more experience and switched to a predominantly training-first 
model. These results suggest that the impacts of WorkAdvance at these 
sites might have been stronger had they been given more time to mature 
before the evaluation started. 


7 Recall that Towards Employment targeted two sectors: health care and manufacturing. Both sectors are 
reflected in targeted sector employment. This analysis does not distinguish the sectors because of sample size 
limitations. 
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A cohort analysis was prespecified as likely to be a key source of variation during the 
analysis planning phase. As mentioned in Chapter 1 , Madison Strategies Group was new to the 
Tulsa area, and Towards Employment had limited experience in the manufacturing sector. It 
was thought that it would take time for these two providers to develop deep relationships with 
employers. In addition, these two providers (Towards Employment and Madison Strategies 
Group) were operating programs that featured a mix of job placement and training first in the 
early intake period and only evolved to a mostly training-focused approach during the second 
half of the intake period. For these reasons, it was expected that the impacts of WorkAdvance 
would strengthen over time. Because of this, it is especially important to analyze whether the 
impacts of WorkAdvance vary by the stage in program development when sample members 
were randomly assigned (that is, by cohort). 

Employment and earnings impacts for the early and late cohorts are shown in Table 6.3. 
The early cohort includes sample members who were randomly assigned before Quarter 4, 
2012, and the late cohort includes the remaining sample members. The sample sizes in Table 
6.3 are relatively small. Therefore, this analysis has statistical power to detect only large differ- 
ences between cohorts. Several patterns in Table 6.3 are noteworthy. 

On the second page, Table 6.3 shows the impacts for the two providers that operated 
mixed training and placement models early in the intake period — Madison Strategies Group 
and Towards Employment. At both sites, there is evidence of stronger effects for the late cohort, 
as expected. At Madison Strategies Group, there are positive and statistically significant impacts 
on both of the survey measures for the late cohort, whereas impacts on these measures were 
close to zero for the early cohort. The estimated difference in the impacts between the cohorts is 
statistically significant for weekly earnings (as indicated by the daggers). In the administrative 
records data, impacts on Year 2 earnings are positive and statistically significant for the late co- 
hort. For the late cohort, Madison Strategies Group increased Year 2 earnings by an impressive 
$4,125 (or 26 percent) above the control group level. In the first quarter of Year 3, the impact is 
no longer statistically significant, though it is still rather large in percentage terms (19 percent) 
and is quite close to the threshold for statistical significance. 

At Towards Employment, there is no clear pattern of cohort differences in the survey 
data, but there is clear evidence of stronger impacts for the late cohort in the administrative rec- 
ords outcomes. Despite control group gains in both outcomes for the late cohort compared with 
the early cohort, Towards Employment produced impacts on both employment and earnings for 
the late cohort. For the late cohort, employment levels in Year 2 were 9 percentage points high- 
er for the WorkAdvance group and earnings levels were nearly $3,000 higher compared with 
the control group. 
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The WorkAdvance Study 
Table 6.3 


Impacts for Subgroups Defined Based on Random Assignment Cohort, by Site 




Early Cohort 


Late Cohort 





Difference 



Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Per Scholas 








Among Year 2 Survey respondents 

Currently employed (%) 

71.9 

58.2 

13.7 ** 

77.5 

70.2 

7.4 


Weekly earnings ($) 

564 

465 

99 ** 

549 

471 

78 * 


Sample size (total = 552) 

152 

132 


135 

133 



Among full research sample, JJI-covered jobs 
Year 1 








Ever employed (%) 

74.9 

67.3 

7.6 * 

79.8 

75.2 

4.6 


Earnings ($) 

8,894 

7,410 

1,485 

8,768 

10,340 

-1,572 

f 

Year 2 








Ever employed (%) 

82.0 

74.6 

7.5 * 

82.5 

78.9 

3.6 


Earnings ($) ] 

17,318 

12,351 

4,967 *** 

18,809 

17,466 

1,343 


First quarter of Year 3 








Ever employed (%) 

70.9 

59.0 

9 ** 

68.7 

71.9 

-3.2 

ft 

Earnings ($) 

5,198 

3,433 

1,765 *** 

5,469 

5,022 

447 

f 

Sample size (total = 690) 

189 

185 


160 

156 



St. Nicks Alliance 








Among Year 2 Survey respondents 

Currently employed (%) 

65.5 

68.9 

-3.4 

63.3 

64.1 

-0.9 


Weekly earnings ($) 

547 

486 

62 

492 

519 

-27 


Sample size (total = 384) 

106 

88 


99 

91 



Among full research sample, JJI-covered jobs 
Year 1 








Ever employed (%) 

73.2 

65.7 

7.6 

83.6 

76.2 

7.4 


Earnings ($) 

8,141 

8,250 

-110 

10,720 

11,442 

-721 


Year 2 








Ever employed (%) 

72.5 

69.5 

3.0 

76.6 

75.4 

1.2 


Earnings ($) ] 

12,482 

12,722 

-240 

16,697 

15,945 

752 


First quarter of Y ear 3 








Ever employed (%) 

56.6 

61.9 

-5.3 

64.4 

65.0 

-0.6 


Earnings ($) 

3,143 

3,619 

-476 

4,910 

4,668 

241 


Sample size (total = 479) 

127 

131 


115 

106 
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Table 6.3 (continued) 




Early Cohort 


Late Cohort 





Difference 



Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Madison Strategies Group 








Among Year 2 Survey respondents 

Currently employed (%) 

71.2 

69.1 

2.0 

82.9 

74.5 

8.4 * 


Weekly earnings ($) 

488 

526 

-38 

605 

525 

80 ** 

tf 

Sample size (total = 560) 

138 

115 


159 

148 



Among full research sample, Ul-covered jobs 
Year 1 








Ever employed (%) 

88.9 

86.1 

2.9 

89.7 

86.9 

2.8 


Earnings ($) 

12,219 

12,514 

-295 

14,286 

13,293 

992 


Year 2 








Ever employed (%) 

78.5 

79.4 

-0.9 

82.3 

85.5 

-3.2 


Earnings ($) 

13,825 

13,902 

-77 

19,549 

15,424 

4,125 *** 

ft 

First quarter of Year 3 








Ever employed (%) 

59.5 

61.0 

-1.6 

70.2 

62.0 

8.2 


Earnings ($) 

3,302 

3,224 

77 

4,396 

3,688 

708 


Sample size (total = 697) 

173 

164 


180 

180 



Towards Employment 








Among Year 2 Survey respondents 

Currently employed (%) 

58.5 

50.7 

7.9 

71.0 

65.6 

5.4 


Weekly earnings ($) 

325 

334 

-9 

435 

385 

50 


Sample size (total = 562) 

144 

142 


142 

134 



Among full research sample, Ul-covered jobs 
Year 1 








Ever employed (%) 

80.8 

75.8 

5.0 

86.6 

81.6 

5.0 


Earnings ($) 

7,537 

7,320 

217 

11,464 

11,549 

-86 


Year 2 








Ever employed (%) 

72.8 

70.5 

2.3 

85.5 

76.3 

9 2** 


Earnings ($) 

9,690 

9,559 

132 

16,598 

13,602 

2,996 ** 

f 

First quarter of Year 3 








Ever employed (%) 

61.4 

59.8 

1.6 

76.6 

67.6 

9.0 * 


Earnings ($) 

2,438 

2,670 

-232 

4,168 

3,593 

575 

f 

Sample size (total = 698) 

168 

177 


181 

172 
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Table 6.3 (continued) 


SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 
insurance administrative records from New York State Department of Labor, Ohio Department of Jobs and 
Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 
includes all sample members randomly assigned in or after Quarter 4, 2012. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 
indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 


These findings might reflect somewhat larger participation differences on occupational 
skills training for the late cohort, which occurred as these providers evolved toward a focus on 
training first and away from a significant placement-first orientation (as discussed in Chapter 
3). 8 Although the sample sizes are relatively small when sites are divided into cohorts (which 
suggests some caution in interpreting these results), the evidence for stronger impacts for the 
later cohort at both Madison Strategies Group and Towards Employment is consistent with ex- 
pectations. 9 

Another result of interest in Table 6.3 is that the impacts in Per Scholas are much weak- 
er in the late cohort. For example, impacts on Year 1 earnings for the early cohort are positive 
and quite substantial. These patterns are very different from the negative (though not statistical- 
ly significant) Year 1 impacts for the late cohort. Impacts on earnings in Year 2 are very large 
($4,967) for the early cohort, but only a third as large (and not statistically significant) for the 
late cohort. It is important to point out that the variation in impacts by cohort at Per Scholas is 
much more obvious in the administrative records than in the survey data. Though the overall 
pattern is the same, much less pronounced differences are evident in the smaller survey sample. 

To understand what might be happening to the late cohort in Per Scholas, it is helpful to 
note that while WorkAdvance group earnings and employment levels do not vary substantially 
between cohorts, the control group employment and earnings levels for the late cohort are much 
higher than for the early cohort. This provides some preliminary evidence that the impacts may 


8 It is not possible to reliably separate out the impact for individuals who were enrolled in the placement- 
first track, because the decision about whether an individual went into that track was made after random as- 
signment. 

9 When the results at Towards Employment and Madison Strategies Group are pooled, the impacts on both 
survey measures, earnings in Year 2, and earnings and employment in the first quarter of Year 3 are statistical- 
ly significant in the late cohort (shown in Appendix Table G. 14). 
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have changed because of the improving local economy or owing to changes in the composition 
of the sample at that site over time in terms of unobservable characteristics. 10 The economy 
probably explains at least part of the difference in control group levels on employment out- 
comes. The New York City economy was rebounding from the Great Recession throughout the 
follow-up period. (This can also be seen at St. Nicks Alliance, another New York City site, 
where control group levels increased by similar magnitudes.) But this difference in control 
group levels doesn’t explain why the WorkAdvance group levels did not increase correspond- 
ingly. An analysis of baseline characteristics across these cohorts shows that the late cohort is 
more likely than the early cohort to be female (18 percent versus 9 percent), is less likely to 
have been currently employed when they entered the study (8 percent versus 1 7 percent), is less 
likely to have been previously incarcerated (4 percent versus 8 percent), and is less likely to 
have been on SNAP (food stamps) (1 1 percent versus 22 percent). 11 But changes in the compo- 
sition of the sample do not appear to drive the patterns in outcomes, as shown in Appendix Ta- 
ble G.5. 12 An analysis of participation data (discussed in Chapter 3) found that, if anything, 
training completion and credentialing differentials were larger in the late cohort. 

The weight of the available data suggests that the economy may have been the main 
factor, but it is difficult to know for sure and other factors may have played a role as well. The 
results may suggest that Per Scholas works better in a weaker economy or that for some reason 
the program was less able to make job placements in the late cohort. Prior research has found 
larger impacts for training programs in weak economies compared with stronger ones. 13 In addi- 
tion, as a mature provider, Per Scholas may not have strengthened its program as much during 
the follow-up period, compared with the other providers. Thus, the impacts at Per Scholas might 
be more sensitive to this countercyclical economic effect. 

The impacts at St. Nicks Alliance are largely the same for both cohorts, despite sus- 
tained and intensive technical assistance at that site. Box 6.2 suggests that St. Nicks Alliance 
may have chosen the wrong sector. The environmental remediation sector has many short-term 
jobs with limited advancement opportunities, and there are long built-in delays between ending 
training and being able to work. In addition, the staff at St. Nicks Alliance never truly commit- 
ted to serving employers as much as the participants. For example, meeting face to face with 
employers is known as a “best practice” for job placement, and St. Nicks Alliance job develop- 
ers seldom were able to get out into the field to do so. 

10 Differences in the observable characteristics of the cohorts do not fully explain the cohort differences. In 
addition, there are no differences in the baseline characteristics of the WorkAdvance and control groups within 
the cohort, so the “bad draw” explanation has been ruled out. 

1 'These results are shown in Appendix Tables A. 1 and A.2. 

12 Appendix Table G.5 shows the results of a regression analysis that interacted treatment status with sites, 
cohorts, and several other characteristics. The results show that the effects for the late cohort at Per Scholas are 
still insignificant after adjusting for several demographic characteristics. 

13 Card, Kluve, and Weber (2015). 
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Box 6.2 


Why Have Impacts Failed to Emerge So Far at St. Nicks Alliance? 

St. Nicks Alliance produced large increases in participation in WorkAdvance services. 
They have operated an environmental remediation training program for over a decade. Un- 
like the Towards Employment and Madison Strategies Group sites, they never operated a 
placement-first model. So why haven’t statistically significant impacts emerged at this site 
so far? 

Several factors may explain the weaker impacts at St. Nicks Alliance. First, demand con- 
ditions in the environmental remediation field were quite weak early in the follow-up peri- 
od, which required St. Nicks Alliance to shift eventually to related industries, such as pest 
control. But demand conditions did pick up in the environmental remediation field later in 
the follow-up period, and impacts are still not evident for the late cohort. The first Work- 
Advance implementation report points to several operational factors that could explain the 
lack of impacts at St. Nicks Alliance.* 

One factor could be organizational structure and background. Unlike most other WorkAd- 
vance providers, the St. Nicks Alliance program operated in the context of a large, multi- 
faceted organization in which workforce development was a relatively small component. 
Senior leaders were not as tuned in to the program, and St. Nicks Alliance had a new di- 
rector, who started just as the evaluation was launching. There were also staffing challeng- 
es: At several points in the evaluation, career navigator positions were understaffed and 
attention was split between WorkAdvance and other programs. Finally, the first report 
notes that St. Nicks Alliance, as a human services organization, was more participant fo- 
cused (and less employer focused) than the other sites. 

St. Nicks Alliance also experienced issues related to their training program, which could 
have reduced the flow of participants through the full program, as shown in Figure 6.2. 
First, because individuals were randomly assigned into cohorts, they often had to wait for 
several weeks before training began. Then the participants who completed environmental 
remediation training to work with asbestos had to wait to take their certification test and 
then wait another 45 days to obtain a “hard card,” which provides city and state authoriza- 
tion to work in the field. That is a long time to wait for a low-income population, many of 
whom may have opted to work elsewhere. 

The first implementation report also notes that job developers at St. Nicks Alliance strug- 
gled to find permanent jobs with advancement potential in environmental remediation, 
which has a heavy concentration of short-term project-based work. St. Nicks Alliance re- 
ceived extensive, ongoing technical assistance to address low placement numbers and 
strengthen job developers’ skills. Meetings with consultants took place weekly, and then 
every other week, for more than a year. For example, job developers often stayed in the 
office rather than meeting with employers in the field. MDRC and consultants worked to 
encourage more proactive job development with stronger direct employer relationships, 

(continued) 
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Box 6.2 (continued) 

such as skill matching between individual participants and job opportunities. St. Nicks Al- 
liance also received assistance in gathering and analyzing systematic labor market infor- 
mation. 

Ultimately there could be many reasons why the impacts were weaker at St. Nicks Alli- 
ance. It is clear that the targeted sector experienced weak demand, and the nature of jobs 
and certifications in that sector may not have enabled longer-term advancement to emerge 
thus far. It is also possible (though the research does not indicate that this is likely, based 
on measurable characteristics, at least) that the weaker impacts so far relate to the charac- 
teristics of the individuals served by the program. 


*Tessler (2013). 


Were the Impacts Affected by the Characteristics of Participants? 

• While some subgroups experienced larger impacts than others, the vari- 
ation in impacts of WorkAdvance from site to site does not appear to be 
explained by variation in participants’ characteristics. 

For the theory of change in WorkAdvance to lead to impacts, the services provided by 
the programs had to target the right population. One possibility that could explain the variation 
in site impacts is that the providers served somewhat different target groups, and it may be that 
some subgroups are more responsive to the program than others. First, a series of subgroup 
analyses looked at the question of whether the impacts of WorkAdvance varied across key sub- 
groups. Then an analysis is discussed that suggests that differences in the compositions of the 
samples from site to site were not a key driver of the variation in impacts. 

Pooled Sample Impacts 

Due to concerns about statistical power at the site level, a decision was made during the 
analysis planning stage to run subgroups using the pooled WorkAdvance sample. Findings for 
the pooled sample are shown in the left panel of Table 6.4. Combining all sites, WorkAdvance 
increased current employment and weekly earnings among survey respondents. WorkAdvance 
increased Year 1 employment, based on unemployment insurance records, by 5 percentage 
points above the control group average of 77 percent. This effect was somewhat surprising giv- 
en that Year 1 was when many WorkAdvance group members participated in occupational 


163 




The WorkAdvance Study 
Table 6.4 


Impacts on Subgroups Defined by Baseline Labor Market Attachment, All Sites Combined 



Full Sample 3 

Fully Attached 

Semiattached 

Long-Term Unemployed 





Difference 



Difference 



Difference 



Difference 


Outcome (%) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 













Currently employed 

70.6 

65.3 

5 3 *** 

82.4 

76.9 

5.5 

68.6 

69.2 

-0.7 

64.6 

53.0 

11.6 *** 

ft 

Weekly earnings ($) 

502 

459 

42 *** 

515 

492 

23 

530 

466 

63 *** 

467 

429 

38 


Sample size (total = 2,058) 

1,075 

983 


254 

287 


373 

344 


445 

348 



Among full research sample 

Year 1 

Ever employed in a Ul-covered job 

82.4 

77.4 

5.0 *** 

93.9 

91.0 

2.9 

82.9 

79.5 

3.3 

74.8 

65.5 

g 2 *h=h= 


UI earnings ($) 

10,299 

10,304 

-5 

14,492 

14,611 

-119 

10,429 

10,233 

195 

7,533 

7,303 

230 


Year 2 

Ever employed in a Ul-covered job 

79.5 

76.7 

2.8 * 

85.8 

89.0 

-3.2 

81.2 

79.5 

1.7 

73.5 

65.4 

g J H=H=H= 

ft 

UI earnings ($) 

15,717 

13,772 

1,946 *** 

18,540 

17,540 

1,000 

16,188 

13,815 

2,373 ** 

13,440 

11,093 

2,347 *** 


First quarter of Year 3 

Ever employed in a Ul-covered job 

66.7 

63.4 

3.3 * 

73.7 

73.6 

0.1 

67.0 

64.0 

2.9 

61.8 

55.5 

6.3 ** 


UI earnings ($) 

4,144 

3,675 

459 *** 

4,457 

4,441 

16 

4,386 

3,671 

715 ** 

3,683 

3,171 

512 ** 


Sample size (total = 2,564) 

1,293 

1,271 


308 

356 


456 

449 


529 

463 
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Table 6.4 (continued) 


SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from 
unemployment insurance (Ul) administrative records from New York State Department of Labor, 
Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

The fully attached subgroup consists of sample members who were working at random 
assignment or who were unemployed for less than one month before random assignment. The 
semiattached subgroup consists of sample members who were unemployed for one to six months 
before random assignment. The long-term unemployed subgroup consists of sample members who 
have never been employed or who were unemployed for seven or more months before random 
assignment. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance 
levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

a Full sample includes all sample members included in one of the subgroups. Three sample 
members are missing a labor market attachment status and are therefore not included in the full 
sample. 


skills training, but recall that two of the providers operated a mixed placement-first and training- 
first model during the early phase of the demonstration. 14 During the second year of follow-up, 
the impact on employment was somewhat smaller. However, the program produced a statisti- 
cally significant $1,946 increase in earnings in Year 2. The same patterns are evident in the first 
quarter of Year 3. 

It is important to note that the pooled impact estimates mask the substantial variation in 
the site impacts. Without Per Scholas, the pooled impacts would be weaker. Without St. Nicks 
Alliance, the pooled impacts would be stronger. Assuming that the WorkAdvance sites repre- 
sent a generalizable set of providers, the pooled impacts provide a good estimate of the impacts 
from an “average” provider. 15 


14 Placement programs typically have impacts more quickly than training programs. See Flamilton et al. 
(2001) for a discussion. 

15 A sensitivity check examined the pooled impacts with and without the Per Scholas site. The results sug- 
gest that Per Scholas drives most, though not quite all, of the positive impacts of WorkAdvance in the pooled 
sample. These results, shown in Appendix Table G.6, indicate that the other three providers produced increases 
in earnings and employment that were much smaller and less consistent across time and measures than the 
impacts at Per Scholas. Excluding Per Scholas, the Year 1 impact on employment is nearly 5 percentage points 
(again, recall that the two placement-first providers are included here). However, the estimated impact on em- 
ployment is statistically significant in neither Year 2 nor the first quarter of Year 3. The Year 1 impact on earn- 
ings is not statistically significant, but the Year 2 impact on earnings is $1,196 and statistically significant at the 
10 percent level. The estimated effect on earnings in the first quarter of Year 3 is not statistically significant 
when Per Scholas is excluded. 
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Confirmatory Subgroup Analysis 

• The impacts of WorkAdvance were concentrated among the medium- 
term and long-term unemployed. Those who entered the program al- 
ready working or who lost their job only within the past month did not 
experience statistically significant impacts on the key economic out- 
comes. At least part of this pattern is due to larger impacts at sites that 
were more likely to serve individuals who had been unemployed longer. 

In WorkAdvance, one confirmatory subgroup was prespecified and is based on sample 
members’ pre-random assignment level of attachment to the labor market. 16 Three groups were 
identified: (1) those who were “fully attached” to the labor market (defined as those currently 
employed or unemployed for less than one month before the study), (2) those who were “semi- 
attached” (defined as those unemployed for one to six months before the study), and (3) the 
“long-tenn unemployed” (defined as those never employed or unemployed for seven or more 
months before the study). The groups were defined this way to be consistent with the categories 
commonly used by labor economists. Generally speaking, the long-tenn unemployed group was 
more distinctive in tenns of participant characteristics than the semiattached and fully attached 
groups. 17 

The expectation was that WorkAdvance would be most effective for the semiattached 
group. This expectation was rooted in the results of previous studies, which showed that em- 
ployment programs can be most effective for those who are at a “tipping point” in their em- 
ployment trajectories. 18 This hypothesis asserts that the fully attached might not benefit from 
WorkAdvance because their outcome levels are relatively high (leaving less room for im- 


16 In recent years, the program evaluation field has become more sensitive to the need to limit the number 
of subgroups analyzed in order to reduce the number of “false positives” that result when one makes too many 
statistical comparisons. (See Box 5.1.) In order to manage this risk, methodologists have recommended pre- 
specification of a limited number of “confirmatory” subgroups that theory and experience suggest might mod- 
erate the impacts of a program (see Bloom and Michalopoulos, 2010). In WorkAdvance, a decision was made 
during the analysis planning phase to focus on subgroups based on prior attachment to the labor market be- 
cause previous studies have found this to be an important subgroup (for example, Friedlander, 1988; Hamilton 
and Scrivener, 2012b). 

l7 The long-term unemployed were older and more likely to be African-American. By definition they had 
lower earnings in the past year, but they also had clearly lower earnings in the three years before random as- 
signment. Per Scholas and St. Nicks Alliance are overrepresented among the long-term unemployed group. 
The semiattached are more likely to be Hispanic compared with the attached and much less likely than the at- 
tached to have entered the program with credentials in the targeted sector. Sample members from Towards 
Employment and Madison Strategies Group are overrepresented among the fully attached group because both 
of these providers were more likely to serve incumbent workers. In addition, the fully attached group had high- 
er educational levels than the semiattached or long-term unemployed. Interestingly, the long-term unemployed 
sample was more likely to be in the late cohorts, perhaps owing to loosening of entry criteria at several sites 
late in the intake period. 

ls Hamilton and Scrivener (2012b); Friedlander (1988). 
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provement) and they may need less assistance from the program than the semiattached and 
long-term unemployed groups. At the other extreme, the long-term unemployed would be ex- 
pected to have more barriers to employment (particularly based on the selectivity of employers 
in the wake of the Great Recession and the resulting scarring from a long duration of unem- 
ployment), 19 which a training-focused program like WorkAdvance might not be equipped to 
resolve. 


As shown in Table 6.4, the impacts of Work Advance tended to be stronger among the 
semiattached and long-term unemployed groups. For the survey outcomes, WorkAdvance pro- 
duced a large and statistically significant impact on current employment for the long-term un- 
employed group and increased weekly earnings among the semiattached group. The estimated 
variation in impacts across subgroups (indicated by the daggers) is statistically significant for 
the current employment measure, but not for the weekly earnings measure. 

The same pattern is evident in the administrative records: WorkAdvance had no statisti- 
cally significant impacts on any of the measures shown in Table 6.4 for the fully attached group. 
Among the semiattached group, WorkAdvance increased earnings by a statistically significant 
margin in Year 2 and the first quarter of Year 3 but did not significantly increase employment in 
either time period. Among the long-term unemployed group, WorkAdvance produced large ef- 
fects on employment and earnings in Years 1 and 2 and the first quarter of Year 3. The estimat- 
ed variation in impacts across subgroups (indicated by the daggers) is statistically significant for 
Year 2 employment. 

Since the sites were pooled together for this analysis, a natural question is the extent to 
which these patterns are driven by the site-specific impacts rather than by the long-term unem- 
ployed group. To check this possibility, a regression analysis tested whether controlling for site 
weakens or eliminates the impacts among the long-tenn unemployed. The results of this test, 
shown in Appendix Table G.5, show that the stronger effects among the long-tenn unemployed 
are eliminated when controlling for site. This suggests that the impacts for this group are at least 
partly due to program-related factors at specific sites. Therefore, the results imply that Work- 
Advance can produce impacts among the long-tenn unemployed, but WorkAdvance does not 
necessarily produce stronger impacts for that subgroup (nor do the results imply that WorkAd- 
vance should not be targeted to the fully attached subgroup). 

Impacts for Exploratory Subgroups 

This section provides a brief summary of the findings from several other subgroup 
analyses. Because these subgroups were not prespecified based on strong theory, these analyses 
should be viewed as exploratory. While these exploratory findings have less standing in the 


19 Kroft, Lange, and Notowidigdo (2013). 
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analysis, they could generate hypotheses for future studies. The exploratory subgroup analyses 
examined the extent to which the effects of WorkAdvance vary by other participant characteris- 
tics, such as age, prior conviction or incarceration status, prior earnings, and education. 

Before reviewing these subgroup results, it is important to highlight that some of these 
findings, like the labor market attachment subgroup findings, could be explained by site-level 
factors. Notably, Per Scholas, which produced larger impacts than the other providers, has a 
younger and more Hispanic sample. In order to help disentangle site-level versus person-level 
characteristics, a compositional analysis was also conducted. This analysis used a variety of in- 
teraction terms and weighting strategies to examine the extent to which the variation in impacts 
of WorkAdvance was attributable to the characteristics of sample members at each site. The 
overall assessment from this analysis, shown and discussed in Appendix G, is that the sample 
composition was not an important factor in explaining the variation in impacts across the sites 
(Appendix Tables G.5 and G.6). 20 Put differently, other aspects of the sites, such as the quality 
of implementation or the economic context in which the programs operated, are more likely to 
be drivers of the variation in impacts. 21 

The overall impression from the exploratory subgroup analysis is that the impacts of 
WorkAdvance did not vary greatly across many subsamples. While there is some variation 
(noted below), in most cases it is not statistically significant. 

• Age: In recent years, there has been increasing concern about the labor mar- 
ket perfomiance of two age groups in particular. First, employment rates 
among young adults (those aged 18 to 24) have been dropping, leading to 
concerns about “disconnected youth.” 22 Another group of concern is older 
workers, many of whom were dislocated by the Great Recession and have 
had trouble regaining a foothold in the labor market. 23 WorkAdvance pro- 
duced statistically significant impacts across all age groups. If anything, im- 
pacts were generally larger among the 1 8-to-24 age group, but this is proba- 
bly an artifact of Per Scholas enrolling more individuals in that age range. 

The variation in impacts across age subgroups is statistically significant for 


20 There is some limited evidence that impacts were stronger for a few measures among sample members 
at the other three sites who have characteristics similar to those of the Per Scholas sample, but results are in- 
consistent across measures. 

_ 1 This conclusion is evident in two ways. First, in Appendix Table G.5, it is clear that site interactions (and 
site-by-cohort interactions) are robust to the inclusion of demographic interactions. In Appendix Table G.6, 
compositional alignment of samples does not seem to coherently explain why Per Scholas’s impacts were larg- 
er or why the other providers’ impacts were smaller. See Appendix G for more discussion. 

22 See Holzer (2010) for a discussion. 

23 See Van Florn, Krepcio, and Heidkamp (2011) for a discussion. 
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only one measure (employment in the first quarter of Year 3). 24 (See Appen- 
dix Table G.7.) 

• Education : While the WorkAdvance programs did a fair amount of screen- 
ing, individuals entered WorkAdvance with varying levels of education. This 
might raise concerns that some entered with insufficient preparation and oth- 
ers with too much preparation for the level of training offered. An analysis of 
the impacts by education levels, however, suggests that there is no pattern ev- 
ident across these subgroups: WorkAdvance produced positive impacts for 
those with lower and higher educational attainment (Appendix Table G.8). 

• Earnings levels : Individuals entered WorkAdvance with different levels of 
prior earnings. Some individuals just barely qualified for the program due to 
higher earnings, whereas others were well below the eligibility cutoff. In 
terms of prior earnings and predicted earnings levels, the estimated impacts 
varied across the subgroups, but not in a consistent way from measure to 
measure (Appendix Tables G.9 and G.10). 

• Prior conviction or incarceration status : One target group of major concern 
is individuals who have a history of criminal convictions or incarceration. 
These individuals often have difficulty obtaining employment, as employers 
in some sectors (such as health care and pest control) are quite reluctant to 
hire individuals with a criminal background. The results bear this out; the 
impacts are somewhat larger among individuals who were not previously 
convicted or incarcerated. Late in the follow-up period, there is statistically 
significant variation in impacts across subgroups based on conviction or in- 
carceration status at study entry (Appendix Table G.l 1). 

• Race : One of the goals of sector programs is to help individuals who have 
historically had trouble entering a sector, either due to discrimination or few- 
er social ties, and to improve diversity in sectors where minorities are un- 
derrepresented. For example, early sector projects focused on helping women 
into construction jobs. In WorkAdvance, generally speaking, impacts are 
larger among Hispanic and black sample members than among white sample 
members. The variation in impacts is statistically significant for earnings in 
Year 1 and employment in Year 2 (Appendix Table G.12). 


24 There is some evidence, presented in Appendix G, that the effects of age are more important when con- 
trolling for other characteristics, such as site and race. The estimate discussed here shows the overall “gross” 
effect of age rather than the “net effect,” which is obtained after one controls for other characteristics. 
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Conclusion 

This chapter has examined three questions to help shed light on the mechanisms behind the im- 
pacts presented in Chapter 5: 

1. Were the impacts driven by either the proportion of participants placed 
in targeted sector jobs or the characteristics of those jobs? 

The weight of the evidence suggests the extent and quality of targeted sector place- 
ments were important factors in explaining the pattern of impacts by site. Per Scholas was by 
far the most successful provider at placing participants in targeted sector jobs. In addition, non- 
experimental evidence suggests that those targeted sector jobs had higher wages than non- 
targeted sector jobs. By contrast, targeted sector jobs at St. Nicks Alliance and Madison Strate- 
gies Group also paid more, but the impacts on getting individuals into those jobs were much 
smaller than those at Per Scholas. At Towards Employment, targeted sector jobs had better 
hours and other non-wage characteristics, but wages were not substantially higher than non- 
targeted sector jobs, and the impacts on getting a targeted sector job were much smaller than at 
Per Scholas. These findings highlight the importance of two factors in strong sectoral programs: 
(1) good job development in the targeted sector and (2) ensuring that targeted sector jobs come 
with a wage premium. 

2. Did the impacts change over time as providers gained more experience 
and moved to a predominantly training-first approach? 

Chapter 5 shows that the overall impacts at Madison Strategies Group and Towards 
Employment were mixed, with some positive impacts on earnings and employment emerging 
late in the follow-up period and some impacts on job characteristics. This chapter gives clear 
indications that the impacts at those sites were stronger for the late cohort, which may reflect the 
movement toward a training emphasis for that cohort and may also indicate that the impacts of 
WorkAdvance improved with the maturity of the programs at those providers (both of which 
were new to sectoral training). It is not possible to reliably disentangle which factor was most 
important. These factors are least relevant to Per Scholas because it was the most mature pro- 
vider coming into the study and had a training-first emphasis in both of its cohorts. In fact, the 
results indicate some weakening in effects for the late cohort at Per Scholas, for reasons that are 
unclear but which may relate to the improving economy. At St. Nicks Alliance, no improve- 
ment in impacts was evident for the late cohort. 

3. Were the impacts affected by the characteristics of participants? 

While there are some subgroups that experienced larger impacts than others, the varia- 
tion in who was served does not seem to explain the pattern of impacts across the sites. One im- 
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portant lesson for the field, however, is that WorkAdvance has shown evidence of improving 
outcomes for those who came into the program already medium- to long-term unemployed. 

This chapter examined some of the possible key drivers behind the variation in impacts 
across the sites. The final chapter will put these results in a wider context by situating WorkAd- 
vance within the literature on demand-driven training programs. 
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Chapter 7 

Conclusion 


The WorkAdvance study provides the first rigorous evidence on whether the promising results 
from the Sectoral Employment Impact Study (SEIS) can be replicated in a different economy, 
scaled up to a wider set of providers, and strengthened through the inclusion of advancement 
services. Since the SEIS results were made public, 1 the sectoral strategy has been implemented 
widely throughout the country and is a key component of the federal Workforce Innovation and 
Opportunity Act passed in 2014. The WorkAdvance evaluation, begun in 201 1 and still in pro- 
gress, has already contributed a great deal of knowledge about how to implement a sectoral 
program with a provider that has not run one before and about how to implement advancement 
services within a sector program. 2 

A year and a half to two years after people applied to WorkAdvance, the impacts of the 
program — the main topic of this report — are promising. Not surprisingly, the most mature 
program, at Per Scholas, a provider with well over a decade of experience operating its sectoral 
program (and which was the only WorkAdvance site also included in the SEIS), had large and 
impressive impacts. Impacts from two of the other three newly established sectoral programs, at 
Towards Employment and the Madison Strategies Group, were less strong, but there were some 
positive impacts on labor market outcomes, particularly in Year 2 and for participants enrolling 
during the second half of the demonstration, who may have experienced a stronger program. 
The remaining WorkAdvance program, at St. Nicks Alliance, shows little evidence of positive 
impacts so far. Participants in that program had the least success in getting jobs in the targeted 
sector. This variation in impacts provides the field with a lot of insights into the conditions 
needed to operate an effective sectoral program. 

In general, the findings suggest that sectoral training programs can produce large im- 
pacts on employment and earnings, but (like any policy intervention) they are not going to be 
effective under all circumstances. Certain conditions need to be met for the strategy to increase 
the employment and earnings of low-income populations, and this chapter aims to shed light on 
what those conditions are. The chapter first summarizes the findings to date from the WorkAd- 
vance evaluation and then places these findings in the context of findings from the SEIS. The 
chapter also discusses lessons from the WorkAdvance study for policymakers and practitioners. 
It concludes with a discussion of what the findings suggest for future tests in this area and out- 
lines some critical unanswered questions for the field. 


'Maguire et al. (2010). 
2 See Tessler et al. (2014). 
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Summary of Findings 

The implementation study shows that mastering new components — whether employer- 
responsive sectoral training, a dual-customer focus, or advancement services — took a substan- 
tial amount of time for all four providers, and for some more than others. The program model 
was designed specifically for the WorkAdvance demonstration, and each provider lacked expe- 
rience with at least one of the WorkAdvance program components, especially postemployment 
services that focused on career advancement rather than only job retention. The providers’ vary- 
ing degrees of experience in each of these areas influenced their implementation of the program 
in different ways. Per Scholas had a sizable head start with its delivery of sector training in in- 
fonnation technology for more than 15 years before WorkAdvance. 

Despite start-up issues, documented in the implementation report, 3 all four providers 
eventually delivered services in all the WorkAdvance model components, with postemployment 
services the last to be developed and implemented. The engagement of participants in key pro- 
gram components — career readiness training and occupational skills training — was high in all 
four providers. In most cases, completion of occupational skills training led to the earning of 
either a nationally or a locally recognized credential, or both. The high participation and com- 
pletion rates may have been, in part, a consequence of the rigorous screening the programs con- 
ducted before enrolling individuals in WorkAdvance (which was also part of the value of the 
programs for employers). 

Notably, WorkAdvance increased participation in occupational skills training at every 
site over what the participants would have gotten on their own, with especially large impacts on 
participation in occupational skills training in the targeted sector — between 38 and 49 percent- 
age points. Many employment and training programs have produced only small or no impacts 
on participation in services. It is not always easy for participants to show up for services, and it 
is certainly difficult for them to finish training programs, yet the combination of WorkAdvance 
services enabled the participants to persevere longer than they would have elsewhere. 

WorkAdvance services cost between $5,200 and $6,700 per participant across the four 
providers. For three of the providers, the range was quite narrow, $5,200 to $5,800, suggesting 
perhaps that many organizations could operate programs similar to WorkAdvance for around 
$5,500 per participant. Roughly half the providers’ operating expenditures were devoted to 
providing preemployment activities and occupational skills training. From a societal perspec- 
tive, what matters is how much more WorkAdvance costs than what society would have spent 
on the participants otherwise. This net cost, which varied from site to site, is what should be 
weighed against the cumulative net change in outcomes, or impacts, when considering the cost- 


3 Tessler et al. (2014). 
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effectiveness of the program. A full benefit-cost analysis is planned after five years of follow-up 
data are available for analysis. 

As discussed above, the impact evidence suggests that an advancement-oriented sec- 
toral training program, such as WorkAdvance, is a promising approach to increasing earnings. 
The impacts here and in the SEIS leave little doubt that the Per Scholas program has large, even 
transformative, effects on participants. Because these impressive results have been measured in 
two rigorous evaluations, policymakers and program operators should carefully review Per 
Scholas’s approach as a model of effective workforce development . 4 

The performance of the other organizations shows that these types of strong results are 
not guaranteed, but the experience of Madison Strategies Group and Towards Employment 
shows how program maturation and a shift toward training can help produce economic impacts. 
Trying to understand the impacts across the sites gave rise to several insights. First, the weight 
of evidence suggests that a critical factor in having an impact was an organization’s ability to 
get a significant proportion of its participants employed in the targeted sector. Second, those 
jobs needed to be good jobs. Providing sector-specific training seemed to be associated with 
being able to secure these better jobs. Given the slow process of developing credibility with 
employers by placing well-prepared individuals in jobs, it is not surprising that these types of 
advancement-oriented sectoral programs seem to take time to reach their potential. 

Per Scholas was by far the most successful site in placing participants in targeted sector 
jobs. In addition, nonexperimental evidence suggests that IT jobs had higher wages than non- 
targeted sector jobs obtained by WorkAdvance group members. By contrast, perhaps due to 
slack demand conditions, St. Nicks Alliance was the least effective site in placing participants in 
the targeted sector, though the jobs in that sector did seem to have higher wages . 5 The pattern at 
the other two sites (like the impacts) was more mixed. Towards Employment and the Madison 
Strategies Group had smaller impacts on placing individuals into sector jobs compared with Per 
Scholas (though these effects were somewhat larger than at St. Nicks Alliance). For Madison 
Strategies Group enrollees, targeted sector jobs paid better and had better benefits and hours. 
For Towards Employment enrollees, targeted sector jobs did not pay much better, but they had 
better hours and advancement characteristics. The overriding factor for site variation in impacts 


4 The Per Scholas impact on earnings was somewhat smaller for participants enrolling in the second half of 
the study, when the economy was stronger, consistent with a commonly seen trend (Card, Kluve, and Weber, 
2015). Control group members enrolling in the later months of the study had much stronger outcomes than 
earlier control group enrollees. It cannot be ruled out that the weaker impacts in the late cohort are due to other 
factors, such as sampling variation. 

5 As mentioned in Chapter 5, targeted sector employment rates were based on the current or most recent 
job. It is possible that the short-term nature of work in St. Nicks Alliance’s targeted sector, environmental re- 
mediation, could have led to an underestimate of the percentage of sample members who worked in that sector. 
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appeared to be differences across the sites in the extent to which they increased placements in 
the targeted sector. 

In the middle of demonstration, two of the programs, Madison Strategies Group and 
Towards Employment, switched from a mixed placement and training model to a training-first 
model. An impact analysis was conducted by cohort — comparing those who enrolled in the 
first half of the study with those who enrolled later — in part to examine the hypothesis that 
sending more individuals initially through training would improve impacts. The impacts, in fact, 
were stronger for the late cohort, but it is unclear whether this was due to the greater training- 
first orientation for the late cohort or to program maturity. The cohort results do not appear to be 
due to changes in sample composition. No improvement through maturity was evident at St. 
Nicks Alliance, which had always implemented a training-first model. 

Finally, while there are some groups of study enrollees who experienced larger impacts 
than others, variation in who was enrolled does not seem to explain the pattern of impacts 
across the sites. Particularly encouraging were results that suggested that WorkAdvance had 
success placing the long-tenn unemployed back into the labor market — a group that is of ma- 
jor policy concern in the wake of the Great Recession. WorkAdvance produced both employ- 
ment and earnings impacts for this group. The model seems particularly effective for the medi- 
um- or long-tenn unemployed, but it is unclear whether it is as effective among incumbent 
workers. This finding echoes other studies of training programs . 6 Because it is hard to disentan- 
gle site effects from demographic characteristics, the program’s effectiveness for these different 
groups should be explored more in the future. 

Where do these results leave the workforce development field? Before coming to some 
preliminary conclusions, it is useful to consider the WorkAdvance findings in light of the find- 
ings for the SEIS, which motivated some core aspects of the WorkAdvance model. 


A Comparison of WorkAdvance and SEIS Findings 

The implementation of the WorkAdvance model can usefully be examined in light of similari- 
ties to or differences from the implementation features of the sector programs in the SEIS. In the 
SEIS, all three providers produced large impacts on earnings. Overall, the pooled impacts of 
WorkAdvance, while positive, are approximately half as large as the impacts of the SEIS. The 
researchers who conducted the SEIS (Public/Private Ventures, or P/PV) suggested that key el- 
ements of the programs included the providers’ experience with sectoral programs, their strong 
relationships with local employers (which they called the “brokering” function), provision of 
job readiness training in addition to occupational skills training, a stringent screening and intake 


6 Card, Kluve, and Weber (2015). 


176 



process, and provision of individualized services. 7 Many of these features were exhibited by the 
WorkAdvance providers, though not all of them and not equally across the providers. 

Provider Experience 

The most important factor in explaining the variation in impacts, perhaps, is that the or- 
ganizational experience of the WorkAdvance providers varied, both among themselves and 
compared with those of the SEIS providers. To be selected for the SEIS, providers had to be 
operating the same program that was eventually evaluated for at least three years. A key differ- 
ence between the two sets of studies, therefore, was providers’ previous experience operating 
sector programs (with the exception of Per Scholas, which participated in both studies) and the 
degree to which they had deep connections in the sector. These connections take time to devel- 
op, even for an experienced organization if it is in a new location, such as Madison Strategies 
Group was in Tulsa. 8 For its part, Towards Employment was new to the manufacturing sector 
and had the compounded challenge of focusing on two very different sectors while coordinating 
program operations in two separate locations. 

Thus, from the perspective of provider experience, the WorkAdvance results are not 
that complicated. The site that was already up and running a sector program (as the SEIS pro- 
grams were) replicated its results well. The three other sites — that were either new to a sector 
approach or new to their locations — had a longer way to go. This “head start” hypothesis as- 
serts that since Per Scholas was running the core elements of the program for a long time and 
already had experience with a random assignment study, it had an insurmountable advantage. 
The results at the other sites suggest that sectoral programs are hard to get off the ground. While 
these results probably explain the impacts to some extent, they don’t explain why a provider 
such as St. Nicks Alliance was successful in producing large participation differences and could 
train people for years in the same field but nevertheless has been unable to produce statistically 
significant impacts on employment so far. (See Box 6.2 for a discussion of why the impacts 
may have been weaker thus far at St. Nicks Alliance.) The value of experience, however, is not 
solely related to running the training program. The benefit comes from working with employers 
in a different way for several years and gaining their trust and respect. The sector “experience” 
hypothesis is given some support by the clear evidence of stronger effects for the late cohort at 
the Towards Employment and Madison Strategies Group sites. 

In order to provide more information to practitioners in this field who are planning on 
starting sector programs, the WorkAdvance research team spoke with leaders at the Per Scholas 


7 Although they aimed to place workers in “good” jobs — jobs that are higher-paying and more stable — 
the SEIS programs did not have an explicit advancement component (Maguire et al., 2010, p. 48). 

s Leaders at Per Scholas report that it takes two to three years before a new Per Scholas affiliate can take 
root and reach full operation in a new location. (See Box 7.1 for more discussion.) 
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and Madison Strategies Group programs. These programs were chosen because both of these 
providers have recent experience in starting sector programs in new locations. The highlights of 
these interviews are discussed in Box 7. 1 . 

Picking the Right Sector 

Chapter 6 presented nonexperimental evidence that Per Scholas’s impacts were largely 
generated by its impressive success in getting people into the relatively high-wage in formation 
technology (IT) sector. In fact, this sector has been booming for some time, and while Per Scho- 
las has had to adapt to changing technology, the sector has not been characterized by the ups 
and downs experienced by, for example, the transportation sector, which is very sensitive to 
cyclical economic factors, or the environmental remediation sector, which is characterized by 
“boom and bust” periods. But this theory — that it was the choice of sector that was critical — 
does not explain why the other sites in the SEIS evaluation also had large wage effects. In addi- 
tion to IT, the SEIS providers worked in many of the same sectors targeted in the WorkAdvance 
evaluation, such as the health care sector (like Towards Employment); manufacturing (like To- 
wards Employment and, later, Madison Strategies Group); and construction (like St. Nicks Alli- 
ance). 9 In addition, the health care sector has been a growth area for some time and has arguably 
been more recession-proof than IT. So if sector matters, how can it be that the SEIS providers 
had larger impacts than the WorkAdvance providers in many of the same sectors? 

It may be that a more textured understanding of what sectors are and how they are de- 
fined is required, one that takes into account providers’ skills and experience and the nature of 
the relationships that providers have with employers in the sector. Broadly defined sectors such 
as “health care,” “IT,” and “manufacturing” cover large swaths of jobs, which range from very 
low- wage jobs with little advancement potential to highly paid jobs at the other end of the wage 
distribution. It may be that the SEIS providers, due to their extensive experience in their sectors 
(including relationships with unions, in one case), were able to find the right subsectors within 
these broader sectors in which to place individuals most advantageously, and to place more in- 
dividuals into those subsectors. The analysis in Chapter 6 found that it was providers’ success in 
placing individuals in targeted sector jobs, combined with those jobs having better characteris- 
tics, that explained site variation in impacts. Long experience in a sector may enable deeper 
webs of relationships to form, which can help providers achieve higher sectoral placement rates 
and may enable providers to find the best niches (or subsectors) within overall sectors. 

In other words, compared with the SEIS providers, the WorkAdvance providers, partic- 
ularly the ones that were newer to the sectors, may have been less able to find the best jobs or 
employers (subsectors) in the sectors in which they operated, particularly early in the demon- 
stration. 

9 Many environmental remediation jobs are in the construction industry. 
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Box 7.1 


How Long Does It Take New Sector Programs to Hit Their Stride? 

One of the main themes of this report is that it takes time for sectoral programs to reach 
maturity and be capable of producing value for program participants beyond what they 
could receive elsewhere. But what is involved in starting up a sector program? In late 2015 
to early 2016, MDRC WorkAdvance evaluators interviewed leaders at Per Scholas and 
Grant Associates, two organizations that have sought to replicate their original sector pro- 
grams in new locations. (Per Scholas has replicated its program in five U.S. cities. Grant 
Associates founded Madison Strategies Group, which operated the Tulsa WorkAdvance 
program, and Madison Strategies Group’s initial leaders had run sector programs in New 
York City.) Even with extensive experience running such programs, leaders of the two or- 
ganizations indicated that it takes between 18 months and three years to get a sector pro- 
gram in a new location to run at full capacity and for participants to have a program expe- 
rience similar to what they would have received in the original program. Presumably, or- 
ganizations new to the sector approach may require even more time. The start-up process 
may go more quickly, however, if an experienced sector program organization already op- 
erates in the location where it starts a new sector program. 

Leaders at Per Scholas, which has run sector programs for 20 years, estimated that it takes 
two to three years before a new provider in a new location is running at the capacity of 
their flagship New York City program. As one leader put it: “You can’t turn on a switch 
and expect everything to be perfect. It takes time to become a known quantity in the com- 
munity.” Grant Associates leaders estimated this time frame to be 18 months. (In Work- 
Advance, this would roughly correspond to the middle of program enrollment for the late 
cohort.) Interviewees emphasized that the goal is not simply to replicate an approach; be- 
cause sectoral programs are inherently local, every program will be different. They also 
spoke of the importance of having local support on the ground to jump-start the relation- 
ships that are so critical to sector programs, and of hiring respected and knowledgeable lo- 
cal staff members to help avoid giving a “carpetbagger” impression. Leaders of both pro- 
viders described a highly systematic expansion approach, driven by both numbers and re- 
lationships, that balances numerous factors, such as local capacity, funding, local need, 
and a highly detailed analysis of the local labor market. As an example, Per Scholas esti- 
mates the target population in a potential new location by conducting an analysis of the 
number of individuals with a high school degree but without postsecondary credentials. 

Leaders stressed the need to take local conditions into account in designing new sector 
programs: adjusting curricula, adding classes in growing subsectors (for example, in data 
security), tracking the number of positions open in different highly specific occupations, 
and understanding the geographic relationships between neighborhoods where potential 
program enrollees live and the locations of the relevant job openings. 

(continued) 
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Box 7.1 (continued) 

The overall impression left by the interviews confirms that a sector program takes time to 
bear fruit. In addition to the considerations above, interviewees noted that success in a new 
community depends on bringing together a long list of actors: local funders, community- 
based organizations, workforce boards, community colleges and training providers, com- 
munity leaders, and, crucially, businesses. Grant Associates and Madison Strategies Group 
leaders felt strongly that the WorkAdvance placement-first track served a critical role ear- 
ly on in building credibility and currency with local employers, although the research sug- 
gests it did not yield strong impacts in the study. 


Connection with Employers 

The SEIS researchers heavily emphasized the role of employer relationships and bro- 
kering as keys to the impressive impacts in that study. As in the SEIS, it was clear that the ser- 
vices at Per Scholas were greatly influenced by employers. Per Scholas has a strong reputation 
and deep relationships with large employers in New York City’s financial and technology sec- 
tor, built up over years. While other providers, notably Madison Strategies Group and Towards 
Employment, worked closely with employers without seeing the same level of impacts, the im- 
pacts for the late cohort indicate that substantial improvement occurred over time. It may be that 
connections with employers and deep specialization in a field take more time to develop than 
the amount of time available in this study. Recall that in the SEIS, each provider had been oper- 
ating for at least three years before the study started. All these threads point in the direction of 
program maturation and evolution as the key driver of site variation in impacts. 

A Focus on Training Rather Than Placement First 

All three providers in the SEIS focused on training as the core intervention. It is possi- 
ble that the placement-first track that was initially operated by Madison Strategies Group and 
Towards Employment did not work well and “depressed” the impacts for these two sites. There 
is clear evidence of strengthening impacts in the late cohorts (which were made up almost ex- 
clusively of training-first participants). In recent discussions, staff members at the Madison 
Strategies Group site emphasized that the early focus on placements was very important be- 
cause the provider was new to Tulsa and had to establish credibility with employers through 
some “early wins.” However, the impacts at these two providers provide clear evidence that 
after the providers moved to a training focus, the impacts were much larger. This is not to say 
that training in and of itself will ensure impacts. The job training field has known for decades 
that training, in order to produce impacts, needs to be demand driven. 
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Overall Assessment 


So where do these various threads lead us? It is evident that sectoral placement matters; experi- 
ence and evolution seem to matter; and a focus on training is important. One key ingredient un- 
derlying these findings seems to be the nimbleness of the provider. While important, a focus on 
training does not seem sufficient in itself; a provider has to provide demand-driven training that 
improves over time according to evolving relationships with employers. As discussed in Chap- 
ter 1 , many training programs have failed to produce impacts when the training provided was 
not nimble and proactive about labor market conditions. Longevity in the field is helpful only if 
an organization is continuously “tapped in” to local demand conditions. 

Given these findings, it is hard to see how a new organization can immediately produce 
impacts. Even the most effective providers need time to leam the sector, to understand how to get 
ahead of trends in the sector, and to calibrate services to meet employers’ evolving needs. The 
impacts of WorkAdvance provide a clear message to the field: With time, providers that are truly 
nimble and demand driven can be expected to produce impacts on earnings — sometimes quite 
large impacts. But these qualities are not to be found in every provider and take time to refine. A 
high priority for the field should be to provide curricula so that more providers can truly internal- 
ize the dual-customer, demand-driven emphasis that research suggests is the active ingredient. 


Considerations for Policymakers and Providers Designing Future 
Sectoral Employment Programs 

The WorkAdvance study highlights that it takes time and a very specific disposition and ap- 
proach to operate an effective sector program. The Per Scholas results show the impressive ben- 
efits that can be attained once a sectoral provider is well embedded in a labor market and func- 
tioning in its prime. The findings here suggest several considerations for designing future sec- 
toral employment programs: 

• Plan for a long start-up period (up to two years) with minimal economic re- 
sults. New sectoral programs require a shift from the rapid job placement ap- 
proach many organizations use for workforce development. Maturity mat- 
ters, particularly for allowing deeper relationships with employers. 

• Carefully study the local labor market at the subsector level. Find niches in 
demand by local employers that are attainable (with training) for participants 
and that offer upward advancement. It is important to verily on an ongoing 
basis that there are jobs available for all graduates of training, and it is also 
important to find niches that most participants cannot access on their own. A 
key difference between sectoral training providers and traditional training 
providers is the ability to stay ahead of, or at least on top of, shifts in demand. 
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• Make sure that these subsectors truly have better wages and advancement 
prospects, compared with what individuals can find on their own. 

• Do not underestimate the level of effort necessary to help individuals through 
training. The WorkAdvance training completion numbers were relatively 
high, but this was probably due at least in part to the substantial effort ex- 
pended by the providers to keep individuals in training. The initial screening 
is also likely to be an important factor. 

• Focus on the funnel through the program, starting with the number of indi- 
viduals who initially walk through the front door. Even at the provider with 
the largest impacts, many either didn’t complete training or completed train- 
ing without finding a job in the targeted sector. 

• Focus on the “wage premium” of the jobs targeted for training. Some provid- 
ers were quite effective at getting individuals into the targeted sector, but the 
evidence suggests that, so far, jobs in some of these sectors had wages that 
were no higher than jobs that WorkAdvance group members found in other 
sectors (though the jobs did have some other beneficial characteristics, such 
as better hours or benefits). 

• Plan for change. This can be changes in employer demand, the local econo- 
my, or other factors. Both the WorkAdvance evaluation and the SEIS found 
that demand shifted at every site. 

• Do not underestimate the importance of soft (or “essential”) skill instruction. 
Employers and participants alike continually highlight the need for and im- 
portance of instruction in these types of skills. 

• Plan to employ staff members who possess the skills needed to operate a 
successful advancement program — strong job developers who get out from 
behind their desks and have close relationships with key employers; staff 
members skilled in career coaching (rather than barrier removal); staff mem- 
bers who have expertise in assessing strong training programs (including 
hands-on ones) and are prepared to adapt curricula as the labor market chang- 
es, and who have very close ties to the needs of employers as well as experi- 
ence in the industry or a willingness to gain it; and leaders who can integrate 
the members of teams, focus on key goals, use metrics to measure progress 
and success, and proactively take corrective action when they see problems. 

A practitioner brief, scheduled to be released in late 2016, will discuss these lessons for practi- 
tioners in more detail. 
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Revisiting the WorkAdvance Theory of Change 

In Chapter 1 , several reasons are mentioned for why WorkAdvance might not produce impacts. 
It is useful to return to those when making an overall assessment of results. The first concern 
was that individuals might not be able to finish training due to a lack of financial support. In 
general, this did not turn out to be an important issue. Training completion rates were high 
across all the sites, and the WorkAdvance group members at every site were much more likely 
to complete training and earn credentials than their control group counterparts. There is still 
room for improvement, as a sizable percentage of sample members did not finish training, but 
low training completion rates do not appear to explain variation in the impacts. 

Another possible breakdown in the theory of change would be if individuals completed 
training but were unable to be placed in the targeted sector. This factor does seem to be central 
to explaining the variation in impacts across sites. Rates of targeted sector placement were high- 
est in Per Scholas (the site with the strongest impacts) and lowest at St. Nicks Alliance (the site 
with the weakest impacts thus far). To some extent this variation is due to different demand 
conditions in the two sectors targeted by these providers, but another reason is likely to be the 
strong relationships Per Scholas had developed with many large employers over years of work- 
ing with them. 

Another factor raised in the program theory discussion was the “sector premium,” that 
is, the expectation that targeted sector jobs would have better characteristics than jobs outside 
the targeted sector. For all four organizations, there were some indications of better job quality 
for targeted sector jobs. However, with the exception of Per Scholas, either the quality ad- 
vantage of the in-sector jobs was not great enough or (more often the case) the organization was 
unable to place enough participants into the targeted sector jobs to translate into large impacts. 

An open question, which will await longer-tenn follow-up, is whether the gains pro- 
duced by WorkAdvance will grow or decay over time. The theory of WorkAdvance holds that 
the placement of people into sectors with viable career pathways as well as the provision of 
postemployment services will lead to a strengthening of impacts over time. In addition, at Madi- 
son Strategies Group and Towards Employment, training was more commonly experienced by 
the late cohorts than by the early cohorts, and some individuals were still in training at the end 
of follow-up for this report. Thus, more time is needed to come to a final assessment of the im- 
pacts of WorkAdvance. Longer-tenn follow-up may also shed light on whether targeted jobs 
need to have higher wages (and not just more opportunities for full-time rather than part-time 
employment) to generate long-tenn impacts. 10 


10 lf WorkAdvance participants (including the control group) are the type of motivated individuals who 
will eventually find full-time employment, sectoral wage premiums could well prove to be key to long-time 
impacts. 
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Conclusion 

Evidence suggests that skill-building in occupations with better jobs can be a means of increas- 
ing earnings in the long run for low-wage workers, as long as training is well aligned with the 
needs of employers, and participants can find employment in the sector for which they were 
trained. Several generations of experiments have also made clear, however, that there are limits 
as to what can be done on the employee side of the equation. Sector programs, in contrast to 
many programs from the past, are heavily demand driven, enabling them to bring workers and 
employers together in ways that solve local and regional economic challenges. The evidence 
suggests that future programs and evaluations should continue to include and examine this 
promising demand-side focus. WorkAdvance has provided the field with important evidence on 
both the potential and the challenges of this approach. The evaluation has highlighted what can 
be accomplished when a provider is deeply embedded in a field and constantly evolving, while 
at the same time underscoring the amount of time and the type of disposition a provider needs to 
have for the strategy to bear fruit. 

WorkAdvance is not the only program currently under evaluation that is designed to use 
a demand-driven skills acquisition approach as a means to advancement for low-income indi- 
viduals. Several programs in the Pathways for Advancing Careers and Education (PACE) 
demonstration use a broadly similar strategy . 11 In addition, evaluations are under way of some 
programs funded through Health Programs Opportunities Grants that use a demand-driven 
training approach to help recipients of Temporary Assistance for Needy Families (TANF) and 
other low-income workers advance in the health care sector . 12 Finally, some programs undergo- 
ing evaluation in the Department of Labor’s Workforce Innovation Fund portfolio use a similar 
strategy . 13 The fact that so many agencies and foundations are operating or supporting programs 
that have evolved in this direction suggests an emerging consensus about the need for a truly 
dual-customer approach to employment programs. In coming years there should be much more 
evidence available to determine the critical features of a successful sectoral program. Is the sec- 
tor placement rate the key to impacts? How much better do the targeted jobs have to be and 
along what dimensions? Are impacts always smaller for incumbent or recently unemployed 
workers, or in stronger economies? Are there other key features not revealed by this study that 
critically affect impacts? Can the strategy work for workers with lower levels of literacy or nu- 
meracy? These ongoing projects have a strong potential to inform workforce policy about the 
scalability of this type of advancement-oriented, demand-driven skills-building approach. 


"Martinson and Gardiner (2014). 
12 Lower-Basch and Ridley (2013). 

13 See U.S. Department of Labor (2015). 
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Wider Policy Implications 

Even if the final results of the abovementioned studies are positive, however, the diffi- 
culty of implementing successful sector-based interventions, coupled with the small size and 
specific focus of some of the models, raises questions about scalability. WorkAdvance, in par- 
ticular, is a challenging model because individual providers have to implement several compo- 
nents simultaneously on their own. An alternative approach, which might aid scalability, would 
be to have different organizations coordinate to implement different components of the model 
— for example, taking advantage of the ability of the community college system to provide 
some program components. (Box 7.2 offers a look at such an approach.) Two sites in the SEIS 
evaluation outsourced training, similar to what was done in three of the WorkAdvance sites, 
suggesting that multiple approaches can be successful; still, careful thought needs to be given to 
how the outsourced training is better than what motivated workers would find for themselves. 
And the Per Scholas results in WorkAdvance suggest caution about this strategy; its ability to 
control all aspects of service provision arguably led to an integrated service experience that may 
partly explain why their program is so successful. Similarly, a smaller provider may be more 
nimble in adjusting to changing employer needs. 

Another challenge with expanding the scale of this strategy is that sector programs are 
inherently small because of the specialization that is necessary to truly understand the high- 
demand niches of the local labor market and to match appropriate individuals to job openings. 
But small programs can become important components of sectoral systems (or “career path- 
ways” systems). In some cities and some labor markets, sector-based programs have been em- 
bedded in much broader initiatives, which also take advantage of feeder systems from “bridge” 
programs to enable a diverse segment of disadvantaged workers to enter the initiative. Project 
Quest, 14 the initiatives implemented by the Instituto del Progreso Latino in Chicago, 15 and the 
Career Pathways system in New York City are examples of programs that apply sector-based 
strategies on a larger scale and in some cases for a more disadvantaged set of workers. 

The WorkAdvance findings have particular relevance to implementation of the Work- 
force Innovation and Opportunity Act (WIOA). The legislation incorporates a strong sectoral 
focus and an emphasis on career pathways, as well as a provision to focus on a disadvantaged 
population similar to that targeted by WorkAdvance. Local Workforce Investment Boards and 
city and state government leaders should consider these findings when putting WIOA into prac- 
tice, and draw upon the lessons herein as they craft requests for proposals and decide how best 
to oversee local workforce providers. 


14 Osterman and Lautsch (1996). 
15 Martinson and Gardiner (2014). 


185 



Box 7.2 


A Collaborative Way to Set Up a Sectoral Initiative: 

The Northeast Ohio Example 

As noted in Chapter 2, Towards Employment is an established community-based organi- 
zation that provides a broad range of employment services for low-income individuals in 
the greater Cleveland area. While it had extensive experience with workforce and work 
readiness programs, it had not operated a sector program before. To operate WorkAd- 
vance, Towards Employment, in contrast to the other three WorkAdvance providers, set 
up a multiagency partnership. Experienced organizations were responsible for delivering 
various aspects of the WorkAdvance services, reflecting the partners’ areas of expertise. 
This allowed Towards Employment to act as a “convener,” and draw on, rather than dupli- 
cate, existing service expertise in the Cleveland area. The involvement of multiple partners 
and providers was also intended to help foster the program’s long-term sustainability. 

Regarding training, for example, Towards Employment “purchased” instruction from local 
community colleges or a private technical school partner to be delivered to cohorts of 
WorkAdvance enrollees, in this way controlling the curriculum content. This arrangement 
also allowed Towards Employment to switch training partners if a provider turned out to 
be more or less cooperative with the WorkAdvance model’s approach or had better or 
worse relationships with employers. As another example, Towards Employment’s use of 
an industry association and a local manufacturing workforce intermediary — as opposed 
to relationships with individual employers — helped it to build sector relationships in a 
new area (manufacturing) more quickly than otherwise would have been possible. These 
intermediaries also provided industry-specific information in such areas as manufacturing 
skills pathways, state and national industry trends, credentialing, labor markets, employer 
engagement, and employer performance appraisal processes. The manufacturing interme- 
diary also helped to identify job openings and match program participants with them. In 
addition to these “employer-facing” partners, Towards Employment contracted with other 
agencies to provide participants with a range of services, including social services and ex- 
tra academic support when needed. 

An arrangement similar to the one used by Towards Employment is probably typical of 
many sector programs around the United States, where there is existing knowledge and 
experience in certain areas and a desire to braid resources and services across different en- 
tities. This type of arrangement, however, is complex to set up and manage, and it requires 
the convener to monitor partners’ performance to ensure they operate in line with the pro- 
gram’s approaches and remain accountable for meeting the program’s specific goals. 


* * * 

These results are not the final word on the WorkAdvance evaluation. Further follow-up is 
planned, which would provide impact results at the three-year and five-year points. 
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Appendix A 


Data Collection, Analysis Methods, and 
Methodological Issues 




This technical appendix describes the collection and analysis approach for the various types of 
data used in this report. The appendix largely follows the structure of the WorkAdvance 
program and the report. It starts with a discussion of recruitment and baseline data. This is 
followed by sections describing the qualitative, program-tracking, survey, cost, and unemploy- 
ment insurance (UI) wage and benefits data. The appendix concludes with a section on the 
impact estimation approach. The remaining appendixes provide supplementary materials for 
Chapters 1 to 6 of the report. 

Recruitment Data 

About a year into the sample enrollment period, MDRC collected data on recruitment and 
intake from each provider. 1 These data were collected to help providers assess their recruitment 
strategies, to help evaluators understand the populations to whom the research results can apply, 
and to understand the process by which participants were identified as eligible for WorkAd- 
vance services. 

Data Collection 

The first step in the process of study enrollment and random assignment was to market 
WorkAdvance and encourage individuals to participate. Each provider collected data on all 
applicants who expressed interest, either on site or by phone, in the WorkAdvance program. 
Staff members recorded how each individual had first heard about WorkAdvance and whether 
he or she was eventually enrolled in the study. If an individual was not enrolled, the provider’s 
staff recorded the step in the intake process during which he or she dropped out or was screened 
out. 2 These data were collected, for each applicant, either in the provider’s management 
information system (MIS) or on a separate paper tracking tool developed by MDRC. Because 
some applicants did not enter the study, the providers shared copies of deidentified data by 
sending them to MDRC via FedEx or by uploading data files to the project’s secure file transfer 
protocol (FTP) website. Upon file retrieval, MDRC stored these files on a secure network which 
restricts access to a limited number of project staff members who manage and work with the 
data. 


Data Analysis Methods 

Using the abovementioned data, MDRC researchers conducted a “funnel analysis” to 
help identify (1) the recruitment sources that yielded the most applicants eligible for the study 
and (2) at what point applicants were selecting or being screened out of study eligibility. 


'Please refer to notes in Appendix Table C.l for further details on the data collection period, by site. 

2 See Appendix Table C.2 for more site-level detail on the intake process, which varied by WorkAdvance 
provider. 
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The analysis tracked interested applicants from their first contact with the program until 
they either dropped out of the intake process or were randomly assigned in the WorkAdvance 
study. 3 Applicants were categorized by their recruitment source and the stage of the intake 
process they were in when they left. 

In preparation for this analysis, the data were securely transmitted to MDRC and 
checked for quality. Data issues identified varied by provider (due at least in part to site-level 
variation in the intake and data collection process): 

• The recruitment data were checked for duplicates (the expected structure of 
the data file was one record per applicant), outliers, consistency across re- 
sponses, and out-of-range dates. No major issues were identified, but MDRC 
made some minor changes to illogical responses (for example, dates with an 
out-of-range year were changed to the year in which the analysis was com- 
pleted). 

• Recruitment data from Towards Employment were available only in aggre- 
gate form, and the data covered a window of time rather than a cohort of ap- 
plicants. To address these issues, MDRC made some adjustments to align the 
counts across intake steps and excluded applicants who were thought still to 
be in the intake process at the time the data report was generated. Also, only 
applicants from the 1 1 most common recruitment sources were included in 
the analysis. 

• Per Scholas began collecting recruitment data at the time applicants attended 
the initial orientation, while the other three sites began collecting recruitment 
data at an earlier point in the intake process. When doing the cross-site analy- 
sis, MDRC did not report on drop-off before orientation for Per Scholas ap- 
plicants. 

• The intake process varied somewhat across the sites, and only common steps 
in the intake process were included in the cross-site analysis. 


3 The first point of contact with the WorkAdvance program varied by site. At Per Scholas, the first point of 
contact was attending the initial orientation. At St. Nicks Alliance and Towards Employment, the first point of 
contact was calling or showing lip in person at the site. At Madison Strategies Group, the first point of contact 
was being scheduled for an orientation. After the first point of contact, data were collected to track the points 
where recruitment numbers dropped off (by site, where available; see Appendix Table C.4), assessment results 
(including eligibility screening), interview results, internal case conference results, and random assignment 
status. Please refer to Chapter 3 of the WorkAdvance implementation report (Tessler et al., 2014) for further 
details on analysis results. 
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Because this analysis was primarily focused on helping providers address recruitment 
challenges and because data were collected for only a small portion of the sample enrollment 
period and were deidentified, these data were not merged with other data used for the study. 
Moreover, no weights were applied to the data. 


Random Assignment and Baseline Data 

Random Assignment Process 

This section describes the procedures WorkAdvance staff members performed when 
enrolling eligible applicants into the study. Random assignment occurred after the recruitment 
and screening process, described in the previous section, was carried out. 

Once interested applicants were deemed eligible for WorkAdvance, staff members ex- 
plained to them in detail the goals of the study, the random assignment process, the data that the 
research team would collect, and the benefits and risks of participation. Next, prospective study 
participants received a WorkAdvance research document packet that contained two fonns: (1) 
the WorkAdvance Study Informed Consent Form and (2) the state waiver authorizing the 
release of UI wage and benefits data for research purposes. 4 Prospective study participants also 
received a verbal explanation from WorkAdvance staff members about the content of each 
form. Study participants needed to sign all consent fonns and undergo random assignment to 
have a chance at receiving WorkAdvance employment and training services. 

After the consent fonns were signed, a WorkAdvance staff member logged into 
MDRC’s online random assignment application and entered the study participant’s identifying 
infonnation and selected (self-reported) measures about the participant’s demographic, educa- 
tion, and employment history. Key identifiers (like name and Social Security number) were 
entered twice into the database. The application recognized discrepancies in data entry of these 
fields and required the WorkAdvance staff member to correct one or both versions before 
submitting the record. Finally, the application randomly assigned the study participant to either 
to the WorkAdvance group or the control group. 5 


4 This release form was administered only at Per Scholas, St. Nicks Alliance, and Madison Strategies 
Group, as the Ohio administrative records agency (which provided UI wage and benefits data for Towards 
Employment sample members) did not require the research study to obtain study participant and notary 
signatures authorizing the release of wage and benefits data. 

5 At the Per Scholas site, for the last 12 months of the sample build-up period, three-way random assign- 
ment was used to help accommodate the provider’s efforts to fill training slots in line with multiple funding 
streams. Per Scholas had a separate grant to serve Supplemental Nutrition Assistance Program (SNAP) 
recipients with WorkAdvance-like services. To ensure that Per Scholas satisfied the sample size goals for the 

(continued) 
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MDRC’s random assignment process uses the widely recommended technique of 
“permuted blocks” in which a block of sample members consists of a given number of control 
and program assignments — usually the number of control assignments equals the number of 
program assignments in each block, for example 10 of each. This is done to ensure an even flow 
of program and control participants, which makes it easier for staff members to manage their 
programs. The order of assignments is sorted by a unique random number associated with each 
assignment. To create a project’s random assignment order several different block sizes are used 
to keep the assignments as unpredictable as possible. Because blocks are constructed of equal 
numbers of program and control assigmnents, the overall sample will be made up of equal 
program (P) and control (C) assigmnents — or nearly so, if the study ends in the middle of a 
block. This method can, in principle, be used for uneven P:C ratios (for example, 60 percent Ps, 
40 percent Cs) as well as designs requiring more than two research categories (such as one-third 
each of P, Q, and C assignments). 

The computer programs that carry out random assignment are implemented in secure 
online applications. Sites are generally provided with their own sequences so that each will stay 
close to the desired P:C ratio. In addition, the occasional site may require smaller block sizes 
than other sites for operational reasons. 

Immediately following random assigmnent, WorkAdvance staff members recorded the 
assigmnent of each WorkAdvance group member into their MIS, which all staff members 
consulted upon engaging individuals in services, to help ensure that control group members 
were not served by WorkAdvance for the full three-year service embargo period. The Year 2 
Survey results, as explained in the main report, suggest that the random assigmnent process 
worked as expected from June 2011 through June 2013, when the study sample was enrolled. 
The treatment (access to WorkAdvance services) and counterfactual (no access) conditions 
were implemented as originally intended by the designers of the evaluation. 

Data Collection 

As described above, MDRC collected a variety of demographic and employment histo- 
ry information on each study participant at the time of random assigmnent. Baseline data 
collection and random assigmnent both ended on June 30, 2013, as expected. 


flinders of the SNAP grant as well as the goals of this evaluation, a three-way random assignment process was 
implemented on July 5, 2012. Individuals who qualified for both WorkAdvance and the SNAP contract were 
randomly assigned to one of three groups: the WorkAdvance program group, the WorkAdvance control group, 
or the “original Per Scholas program group,” whose members were offered services similar to WorkAdvance 
services (minus postemployment support), funded by the SNAP grant. 
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To collect this information, provider staff members primarily used an MDRC online 
baseline data collection tool. To help ensure data quality, the online baseline collection tool 
imposed certain rules for accepted values, controlled skip patterns (where a response might lead 
to additional questions), and prompted users with various messages if they reported potential 
errors or did not answer all the questions. If the online random assignment system was not 
functioning for any reason, staff members collected baseline information on a paper form that 
was later sent to MDRC for data processing. In this scenario, provider staff members were 
instructed to review the baseline forms to check for the types of issues the online tool controlled 
for automatically, but the manual process was somewhat more error prone. In total, around 98 
percent of the study sample members had their data entered into the online tool, with the 
remaining 2 percent having data recorded via the paper tool. The quality of these baseline data 
was high overall but was somewhat lower for baseline data collected via the paper form. 

Data Analysis Methods 

In general, the strict validation rules built into the baseline data collection system great- 
ly reduced the number of data analysis issues encountered. Still, MDRC routinely ran quality 
control procedures on the baseline data, to check for the following: 

• Reasons to exclude enrolled participants from the impact analysis. 6 Aside 
from withdrawing from the study, 6 7 participants could be excluded from the 
impact analysis for a few reasons: being served under another funding stream 
(N = 22), missing a signed consent form (N = 1), or having been enrolled be- 
fore the official start of random assignment (N = 5). 

• Systematic problems with out-of-range, missing, and outlier response values. 

Only minor issues of this type were found, which appeared to be driven pri- 
marily by keystroke errors and misunderstood questions. MDRC periodically 
provided technical assistance to the provider staff members, to help ensure 
the quality of responses over time. 

• Logical and consistent item response patterns across research groups. No 
major issues of this type were found. 

MDRC conducted a special analysis to check for statistically significant differences in 
select baseline characteristics by research group. As expected, only a few statistically significant 
differences in baseline characteristics were found. In addition, a logistic regression was run to 


6 All sample members enrolled through Compass, a provider in northeast Ohio, were excluded from the 
impact sample when the site was dropped from the study. 

7 A total of 20 participants withdrew from the study after random assignment. 
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test whether key baseline characteristics could predict whether a participant was in the program 
group, both for the full sample and at each site. The model included 21 covariates that were 
regressed on a program group indicator (P = 1 = program group; P = 0 = control group). Neither 
the cross-site model (p-value = 0.755) nor any of the site-specific models are statistically 
significant, indicating that program and control group members do not differ significantly 
across the key baseline characteristics. 

MDRC also conducted a special analysis to check for statistically significant differences 
in select baseline characteristics by random assignment cohort (that is, when participants were 
enrolled). The results of this analysis, which are presented in Appendix Tables A.1 and A.2, 
suggest that — for each of the four WorkAdvance sites — study participants who enrolled in 
the first half of the sample build-up period are different from those who enrolled in the second 
half. 8 


Qualitative Data 

Qualitative Data Sources 

Focus Group and Participant Interview Data 

Focus groups were conducted once for each sector in the program: one focus group 
each at Per Scholas, St. Nicks Alliance, and Madison Strategies Group, and two focus groups at 
Towards Employment for the health care and manufacturing sectors. The focus groups were 
conducted in Quarter 3, 2013, two years after sample enrollment began at Per Scholas, 20 
months after sample enrollment began at St. Nicks Alliance, and 19 months after sample 
enrollment began at Madison Strategies Group and Towards Employment. The focus groups 
were made up of 8 to 12 WorkAdvance group members who were interested in participating, 
with preference given to individuals in a shared training cohort who were close to the end of 
their occupational skills training. Participants were infonned of their option to participate in 
advance. Participants in the focus groups were each served food and given $25 cash as a token 
of appreciation for taking the time to speak with MDRC researchers. They were asked about 
their motivations for applying to the program, their experiences with WorkAdvance skills 
training and other services, and their general impressions of the program. 


s Study entry year is included as a covariate. More details on covariates can be found in the last section of 
this appendix. 
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The WorkAdvance Study 
Appendix Table A.l 


Selected Characteristics of Research Sample Members at Baseline, by Random Assignment Cohort and Site 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Characteristic 

Early 

Cohort 

Late 

Cohort 

Early 

Cohort 

Late 

Cohort 

Early 

Cohort 

Late 

Cohort 

Early 

Cohort 

Late 

Cohort 

Demographic characteristics 

Female (%) 

9.1 

18.0 *** 

14.8 

14.1 

18.1 

13.9 

64.6 

52 7 *** 

Average age (years) 

31 

31 

35 

36 

37 

23 *** 

37 

34 *** 

Race/ethnicity (%) 

Latino/Hispanic 

33.5 

38.9 

19.5 

26.8 * 

4.8 

7.2 

4.1 

5.1 ** 

White 

6.2 

4.1 

7.2 

6.6 

37.9 

40.4 

14.2 

22.5 

Black/ African-American 

45.4 

43.4 

67.7 

56.8 

29.6 

27.3 

75.3 

66.4 

Other race 2 

14.9 

13.6 

5.6 

9.9 

27.8 

25.1 

6.4 

6.0 

Parent of one or more children (%) 

27.8 

24.4 

51.6 

38.0 *** 

51.6 

51.9 

49.6 

54.0 

Education level (%) 

Highest level of educational attainment 

Less than GED certificate or high school diploma 

0.3 

0.0 

15.9 

y 2 *** 

6.8 

5 3 *** 

11.9 

0 o *** 

GED certificate/high school diploma 

38.5 

35.4 

41.1 

48.4 

30.9 

40.3 

34.5 

39.7 

Some college 

34.0 

30.7 

27.5 

25.3 

48.7 

47.5 

41.4 

52.4 

Associate's degree/2-year college 

8.3 

11.7 

5.0 

10.4 

7.1 

4.4 

6.1 

3.7 

4-year college degree or more 

19.0 

22.2 

10.5 

8.6 

6.5 

2.5 

6.1 

4.2 

Already has a license/certificate in targeted sector 

5.1 

j g ** 

3.5 

0.0 [***] 

17.0 

10.3 *** 

23.8 

26.1 

EmDlovment status 

Number of months of current unemployment spell (%) 

Never employed 

4.0 

3 2 *** 

2.7 

1.4 

0.6 

0.8 * 

2.9 

2 5 *** 

Currently employed 

17.4 

7.9 

9.8 

11.3 

28.5 

25.0 

17.7 

35.4 

12 months or less 

50.8 

62.3 

54.1 

57.9 

53.1 

62.5 

43.8 

44.2 

More than 1 2 months 

27.8 

26.6 

33.3 

29.4 

17.8 

11.7 

35.7 

17.8 
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Appendix Table A.l (continued) 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 


Early 

Late 

Early 

Late 

Early 

Late 

Early 

Late 

Characteristic 

Cohort 

Cohort 

Cohort 

Cohort 

Cohort 

Cohort 

Cohort 

Cohort 

Number of months in current or most recent job (%) 

12 months or less b 

57.0 

53.8 

58.0 

49.3 * 

60.8 

70.6 *** 

55.4 

61.2 

More than 12 months 

43.0 

46.2 

42.0 

50.7 

39.2 

29.4 

44.6 

38.8 

Is or has been employed in targeted sector (%) 

7.8 

8.0 

2.4 

1.9 [ ] 

17.0 

12.6 

24.3 

36.7 *** 

Average hourly wage at current or most recent job c ($) 

11.99 

11.80 

12.59 

13.47 

10.31 

10.33 

9.68 

10.07 

Worked full time (35 or more hours per week) d (%) 

58.5 

58.8 

64.7 

65.6 

70.7 

72.0 

61.5 

60.8 

Circumstances that mav affect iob chanse or retention (%) 

Previously convicted of a crime 10.7 

10.1 

26.1 

12 7 *** 

39.1 

40.3 

22.9 

27.8 

Previously incarcerated 

8.0 

4 i ** 

25.6 

g 5 *** 

36.3 

32.2 

9.3 

14.7 ** 

Income 

Average monthly family income ($) 

714 

568 ** 

665 

730 

860 

753 

471 

727 *** 

Income sources (%) 

Food stamps/SNAP 

22.5 

114. *** 

39.5 

44.8 

35.9 

33.1 

64.1 

46.5 *** 

Welfare/TANF 

6.2 

6.3 

15.0 

12.3 

0.6 

0.8 [ ] 

5.2 

2.5 * 

Unemployment insurance benefits 

25.4 

23.1 

27.8 

20.9 * 

9.2 

5.8 * 

11.9 

9.1 

Sample size (total = 2,564) 

374 

316 

258 

221 

337 

360 

345 

353 


SOURCE: MDRC calculations from the WorkAdvance baseline information form. 

NOTES: GED = General Educational Development; SNAP = Supplemental Nutrition Assistance Program; TANF = Temporary Assistance for Needy Families. 

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort includes all sample members randomly assigned in 
or after Quarter 4, 2012. 

Chi-square tests were used for categorical variables and t-tests were used for continuous variables to assess the difference in average characteristics between 
program and control group members. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Square brackets 
indicate the chi-square test may not be valid due to small sample sizes. 

Sample sizes may vary because of missing values. 

a "Other race" includes sample members who identify as non-Hispanic and listed "Asian," "American Indian," or "Other" as their race, including sample 
members who answered "multiracial." 

b Measure includes sample members who have never been employed. 

c Wages for sample members who have never been employed are counted as $0. 

d "Worked full time" does not include sample members who have never been employed. 



The WorkAdvance Study 
Appendix Table A.2 


Estimated Regression Coefficients for the Probability of Being in the Early Cohort 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Characteristic 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

Female 

-0.745 *** 

0.257 

0.228 

0.610 ** 

Black 

-0.111 

0.376 

0.183 

0.417 ** 

Hispanic 

-0.384 * 

0.332 

-0.339 

0.097 

Age 

-0.012 

-0.019 

0.040 *** 

0.031 *** 

Under age 24 

-0.267 

-0.812 ** 

0.436 * 

0.302 

Number of children living with respondent 

0.285 ** 

0.143 

-0.027 

-0.080 

Born in state 

0.427 * 

0.051 

0.117 

0.135 

Highest level of education 

-0.132 ** 

NA 

NA 

NA 

Highest level of education is GED or less 

NA 

0.511 ** 

NA 

NA 

Highest level of education is 2- or 4-year degree or more 

NA 

NA 

0.679 ** 

NA 

Highest level of education is high school, GED, or less 

NA 

NA 

NA 

0.335 * 

Already has a license/certificate in targeted sector 

0.779 

14.840 

0.337 

-0.053 

Currently employed at baseline 

1.085 *** 

-0.219 

0.161 

-0.408 * 

Monthly family income 

0.000 

0.000 

0.000 

0.000 

Employed in at least 1 quarter in year prior to RA 

0.153 

-0.664 ** 

-0.334 

-0.323 

Employed in all 4 quarters in year prior to RA 

-0.275 

-0.647 * 

-0.232 

-0.489 * 

Number of quarters employed in 3 years prior to RA 

-0.043 

0.166 *** 

0.008 

-0.038 

Total earnings in quarter prior to RA 

0.000 

0.000 

0.000 

0.000 

Total earnings in 3 years prior to RA 

Received unemployment insurance benefits in 

0.000 * 

0.000 

0.000 

0.000 

3 years prior to RA 

-0.242 

-0.568 ** 

-0.098 

NA 

Randomly assigned to environmental remediation sector 

NA 

2.094 *** 

NA 

NA 

Randomly assigned to health care sector 

NA 

NA 

NA 

-0.171 

R-squared 

0.095 

0.264 

0.080 

0.123 

Chi-square statistic 

68.549 

146.823 

58.443 

91.849 

P-value of chi-square statistic 

0.000 *** 

0.000 *** 

0.000 *** 

0.000 *** 

Sample size (total = 2,564) 

690 

479 

697 

698 
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Appendix Table A.2 (continued) 


SOURCES: MDRC calculations from the WorkAdvance baseline information form and 
unemployment insurance administrative records from New York State Department of Labor, Ohio 
Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: NA = not applicable; GED = General Educational Development certificate; RA = random 
assignment. 

The early cohort includes sample members randomly assigned through Quarter 3, 2012. The late 
cohort includes sample members randomly assigned in or after Quarter 4, 2012. 

A chi-square test was applied to differences between outcomes for the early and late cohorts in the 
analysis for the full research sample. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 


During the focus groups, a total of 20 interested participants were randomly selected to 
be interviewed individually as well: four each from Per Scholas, St. Nicks Alliance, and 
Madison Strategies Group; and four each from Towards Employment’s manufacturing and 
health care programs. MDRC researchers interviewed all these participants in Quarter 3, 2013, 
and contacted these same participants in Quarter 4, 2013, for follow-up interviews. In addition, 
informal case reviews were conducted for each of these participants with the staff members 
most familiar with them. 

Ultimately, 18 of the original 20 participants were able to be interviewed in person 
again three months later (Quarter 4, 2013). In-person interviewees were given $25 cash as a 
thank-you; phone interviewees were sent $25 gift cards after the interviews. Interviewees were 
asked about their motivation for applying to the program, interest in the sector, plans for the 
future, and experiences with WorkAdvance services, and for feedback about the program. 

Sample interview questions for participant focus groups and participant one-on-one in- 
terviews are included in Appendix Box A. 1 . The convenience sampling and small sample sizes 
for focus groups and participant interviews limit the generalizability of findings from these 
sources. 


Employer Interviews 

Phone interviews with employers of WorkAdvance group members were conducted in 
Quarter 4, 2013. MDRC researchers worked with WorkAdvance providers to identify individu- 
al staff members at employers who could comment on the relationship between WorkAdvance 
and their finns. MDRC interviewed a total of 21 employers: four with connections to Per 
Scholas, six with connections to St. Nicks Alliance, five with connections to Madison Strategies 
Group, and six with connections to Towards Employment — three health care employers and 
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Appendix Box A.1 

Sample Interview Questions for Participant Focus Groups 
and One-on-One Participant Interviews 

I. Interest and Expectations 

1. Why did you sign up for this program/training? 

a. What did you hear about [WorkAd vance program] that got your attention? What 
attracted you to the program? (Probe: Was the fact that it was free the main thing 
that stood out? Did you hear that you could earn more money by participating in 
this program? Did you hear you could establish yourself in a new career?) 

b. What did you hope/expect to get from the program? We are not asking what you 
hope the result will be of your participation in this program, but what kinds of 
skills or services did you hope to get from the program. Please be specific. 
(Probes: specific occupational skills, help with your resume, job leads, career 
coaching.) 

c. Did you consider other ways of getting these skills or services? If so, what ways 
or where? 

2. Were you specifically looking to work in the industry when you first ap- 

plied? Why? 

a. What interests you about this sector? 

b. What did you know about the industry and the types of jobs that are 

available in that sector before you started training? Were there specific jobs you 
were hoping to be able to get? Which ones? 

c. For those of you who were not specifically interested in , what inter- 

ested you about this training/program enough to sign up anyway? 

3. This program tries to help people not only find a job, but also plan to move ahead in 
their jobs or careers later on. What does “moving ahead” or “moving up” in a job or 
career mean to you? (Probes: Getting a better-sounding title? Earning more money? 
Getting better hours? Getting better benefits? Becoming a supervisor?) 

II. Occupational Skills Training 

1. Please describe the occupational skills training you are participating in. We are not re- 
ferring here to general life skills or job readiness skills training, but to the sector- 
specific skills training. 

(continued) 
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Appendix Box A.1 (continued) 

a. What type of training program was/is it? What were/are you being trained to do? 
What type of job are you expecting to look for? 

b. Please describe a typical day in this training class. 

i. How did you spend your time? 

(Probes: Did you spend most of your time in or outside of a classroom? 
Did you spend most of your time sitting at a desk and listening to a teacher 
or working hands-on to learn new skills — for example, taking apart a 
machine? Did you work individually, or were there group projects? Was 
there a lot of reading? Did you feel comfortable asking questions?) 

ii. What did you like or dislike about how the training content was delivered? 

c. Did you visit any employers or work sites? Did employers come to visit your 
class and talk about what types of jobs might be available? What kinds of things 
did you leam from those visits? 

d. We understand that you were in training with (read whichever applies): other 
members of this program (including others in this focus group) / other people in 
the general population — not necessarily with people from [provider] / a mix of 
people, some from [provider] and some not. What did you like or dislike about 
this arrangement? 

e. While you were in training, was there any discussion about career or job planning 

beyond getting your first job in the sector? Please describe who started 

and participated in those discussions and what you talked about. 

f. During the occupational skills training, were you in touch with the program staff 
from [provider]? In what ways? (Did they visit the training site? Did you receive 
phone calls or e-mails? How frequently? Did you visit the program offices?) 

g. Besides paying for training, what other forms of assistance did the program pro- 
vide? (Probes: transportation vouchers, tools, equipment, books, licensing fees, as 
well as other supports from the staff.) 

h. Were there times during your training program when it was difficult to partici- 
pate? What made it difficult? How did you handle those situations? Did you ever 
consider dropping out altogether? If yes, what made you feel that way? How did 
you handle that and stay in the training? 

i. [High Priority] Do you feel that the training you received through [provider] was 
different from the kind of training you could have received through another pro- 
gram? Or if you have ever participated in other training programs, was this train- 
ing different? If yes, how? 

j. What would you change about the occupational skills training, if anything? What 
do you think worked well? 
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three manufacturing employers. The interviewees were asked how they learned about WorkAd- 
vance, their experiences with WorkAdvance participants, and their relationships with the 
WorkAdvance providers. Sample interview questions for employers are included in Appendix 
Box A.2. 

Staff Interviews and Program Observations 

Face-to-face and phone interviews with program staff members were conducted at vari- 
ous points over the study period. MDRC researchers collected infonnation on staff members’ 
understanding of the goals and purposes of WorkAdvance; recruitment and enrollment; devel- 
opment and delivery of the specifics of the coaching, skills training, and career readiness 
training components; and participant motivation and engagement. As an example, Appendix 
Box A.3 presents a list of selected staff interview questions that were asked regarding the 
postemployment component of WorkAdvance. 

MDRC researchers and consultants conducted site visits and observations of compo- 
nents of the program throughout the study. This research included observations of intake 
processes and orientations at Per Scholas, St. Nicks Alliance, and Madison Strategies Group in 
Quarters 2 and 3, 201 1; career readiness training at Towards Employment and Madison Strate- 
gies Group in Quarter 3, 2013; career coaching at Per Scholas in Quarter 1, 2014, at Madison 
Strategies Group in Quarter 2, 2014, and at Towards Employment in Quarter 3, 2014; and 
postemployment alumni workshops at Per Scholas in Quarter 1, 2014. 

Provider-Created Program Materials 

During site visits, MDRC researchers collected samples of materials used by the pro- 
viders in their implementation of WorkAdvance. These materials included curriculum docu- 
ments for career readiness training at all four sites, recruitment materials such as flyers, and 
scripts for career coaching sessions. These materials were analyzed to help MDRC researchers 
understand the content, length, and intensity of the career readiness training; methods and 
messaging for recruitment; and the content of the career coaching component of the program. 

CEO Social Innovation Fund Quarterly Progress Reports 

MDRC researchers drew some descriptive information from quarterly grant reports that 
the WorkAdvance providers submitted to New York City’s Center for Economic Opportunity 
(CEO). 
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Appendix Box A.2 

Sample Interview Questions for Employers 

I. Questions Geared Primarily Toward More-Involved Employers 

1. How did you first hear about (PROVIDER)? Did someone refer you to (PROVIDER) 
or did (PROVIDER) staff reach out to you directly? What message did you first hear 
about (PROVIDER) that made you think it might be worth pursuing a relationship? 

2. What were you hoping to achieve by working with (PROVIDER)? 

3. A primary goal of (PROVIDER) is to be sure that the training and other services they 

provide to eventual job applicants in the sector are aligned closely with the 

needs of local employers in the sector. 

a. Has staff at (PROVIDER) ever actively sought your company’s input or 
guidance about the training or other activities they conduct — for example, 
the way they screen applicants for their training program, the technical or 
soft skills curriculum they use, etc.? If yes, which activities have they sought 
input on? 

b. Has your company ever had the opportunity to provide this kind of input or sup- 
port to (PROVIDER)? If yes, please describe. 

[Probe, if necessary]: Other types of guidance or input they might seek could in- 
clude: 

i. Selection of targeted occupations and description of career paths 

ii. Placement services 

o Visit training classroom to describe jobs and workplace environment 
o Employer site tours 
o Mock interviews 

o Work experiences such as internships or mentoring 

4. What kinds of services have you received from (PROVIDER)? 

5. Did (PROVIDER’S) services help you and your company in any of the following 
ways? 

i. Reduced time to hire 

ii. Reduced turnover 

iii. Troubleshooting with new hires — for example, assistance to: 
o Improve attendance 

o Improve j ob performance 

iv. Reduced safety infractions 

v. Help filling skilled positions beyond entry level 

6. [If employer has hired a relatively large number of WA graduates]: Have any of these 
services saved your business money or helped your business grow or expand? 

(continued) 
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Appendix Box A.2 (continued) 

7. Did (PROVIDER’S) services to your company meet your company’s expectations? 

8. How does the experience of working with (PROVIDER) compare with the experience 
of working with your other recruitment sources or other staffing agencies? 

a. Is (PROVIDER) an important source of recruitment for your company when 

you are trying to fill jobs in the industry/occupation? 

b. How do you evaluate if these recruitment sources are effective in terms of 
providing qualified workers? 

c. Do you expect to hire again from this provider? 

d. Would you recommend them to other employers? 

9. Are there any other ways in which your company benefited from the services it re- 
ceived from (PROVIDER)? 

10. How could (PROVIDER) have improved your company’s experience with them? 

a. [if provide feedback:] Did you share any of that feedback with (PROVIDER)? 

11. Are there any services (PROVIDER) could offer to your company to be helpful in 
meeting your needs as a (COMPANY INDUSTRY [ex: trucking provider])? What 
kinds of services? 

a. [if yes:] Have you ever shared that feedback with (PROVIDER)? 

12. Are there any ways that (PROVIDER) could better prepare individuals for jobs with 
your company or in your industry more broadly? If yes, how? 

a. [if yes:] Have you ever shared that feedback with (PROVIDER)? 

13. WRAP-UP QUESTION: Are there any other ways that you’ve interacted with 
(PROVIDER) that we haven’t touched on here today? 


Qualitative Data Analysis 

Qualitative data collected for the WorkAdvance evaluation were analyzed and are pre- 
sented throughout the implementation and impact reports to give context to the program and 
describe how WorkAdvance was implemented. The interview and focus group data provide rich 
information that added to the quantitative baseline and participation analyses. Interview tran- 
scriptions and notes were coded into cross-cutting themes and trends. Codes were developed 
based on key research questions, emerging themes and lessons, and other topics of inquiry. 
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Appendix Box A.3 

Selected Interview Questions for Provider Staff Members 
Regarding Postemployment Component 

1 . In your interactions that involve individualized communication, what do you usually 

discuss? 

a. Can you give me an example of some of the points you touch on, or how a typical 
post-employment conversation might flow? 

b. What do you do to prepare for those conversations, if anything? 

c. Do you discuss or review the Individual Career Plan (ICP)? 

d. Do those conversations produce action items or next steps? What are they? 

e. How does the medium of conversation (phone, email, etc.) change the content of the 
discussion? 

f. After those conversations, do you send any written record of what was discussed? Is 
the ICP updated in each conversation with the participant? If so, does the participant 
receive a copy of the updated ICP? 

2. How is follow-up conducted after advancement conversations? 

a. What are some of the challenges participants face in following through with next 
steps and advancement action items? 

b. Do you use incentives to motivate participants to follow through on the next steps or 
to reach specific milestones? 

3. Does anyone else reach out to or stay in touch with participants postemployment that you 

haven’t yet mentioned? 

a. To that end, how do staff coordinate and communicate their respective participant in- 
teractions? 


Sample codes included basic program features such as “career readiness” or “occupational skills 
training” and implementation processes such as “recruitment,” while other codes explored 
trends, such as “interest in sector” and “institutional context.” Coded interview excerpts were 
then aggregated and analyzed accordingly. 
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Program-Tracking Data 


Quantitative Program-Tracking Data Collection 

Staff members at all four providers recorded information for each WorkAdvance group 
member on his or her participation in the program between June 2011 and December 20 14. 9 
Staff members tracked participation in activities in their own MIS or using paper tools MDRC 
designed for special analyses. These data were securely transferred (and stored on a restricted 
access network) to MDRC every four to six months. The program-tracking data presented in 
this report cover WorkAdvance group participation in services and activities within the first 1 8 
months of program entry. Provider staff members recorded data on career readiness services, 
supportive services, occupational skills training, placement (not shown in this report), and 
career advancement coaching services. 

Before collecting these data, MDRC conducted an extensive MIS assessment with each 
of the providers to evaluate system capacity to reliably track participants’ engagement in 
WorkAdvance services and activities. This assessment comprised a series of discussions with 
program staff members and data system administrators about the WorkAdvance service flow 
and the procedures for recording and managing information about the providers’ engagement 
with participants along the way. Data systems and paper case files were also reviewed to check 
for consistency and completeness across information sources. 

Data Analysis Methods 

Using program-tracking data, MDRC analyzed program participation for all WorkAd- 
vance group members. The analysis covered program activities that occurred between June 
201 1 and December 2014. 

Upon receipt of the data files, MDRC performed its standard data quality procedures, 
which included checking for missing, out-of-range, outlier, and overwritten data. Provider staff 
members routinely helped MDRC resolve data issues identified in these files to help facilitate 
an accurate interpretation of the participation in WorkAdvance services. Further, for the purpose 
of making an overall assessment of how well these data represent the services program partici- 
pants received, MDRC checked each provider’s data with a few questions in mind: 


9 The WorkAdvance sites began enrolling individuals in the study in different months: Per Scholas began 
in June 2011; St. Nicks Alliance began in September 2011; and Madison Strategies Group and Towards 
Employment began in October 2011. All sites completed study enrollment at the end of June 2013. 
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• Are the activity trends reasonable relative to the provider ’s planned service 
sequence? For this purpose, where possible, MDRC compared activity rates 
among participants with different lengths of follow-up. 10 

• Are activity rates the same as or similar to the provider’s monthly perfor- 
mance monitoring reports? This is a measure of whether the providers and 
MDRC had the same concept of how to define and measure participation in a 
WorkAdvance activity. 

• Is the provider’s account (as recorded in the MIS) the same as or similar to 
the report made by the participant in the follow-up survey regarding what 
she or he did in the program? There are often some differences between 
what a program-tracking system will show and what an individual may recall 
or think of during a survey interview when asked specific questions about 
what she or he did. Additionally, WorkAdvance participants may have re- 
ceived services outside the WorkAdvance program. Because of this, pro- 
gram-tracking data and survey data are not expected to match completely, 
but this type of comparison does provide some measure of the extent to 
which provider staff members use their systems to record participant activi- 
ties. Appendix Table A.3, which illustrates the extent to which training par- 
ticipation rates varied between the MIS and survey data, shows that the pro- 
viders’ training records are largely consistent with WorkAdvance group 
members’ responses to the question about their participation in sector train- 
ing. For example, 83 percent of Towards Employment WorkAdvance group 
respondents indicated that they either did (56 percent) or did not (27 percent) 
participate in training in the targeted sector, and the provider’s MIS data sup- 
port these responses. The remaining 17 percent of the site’s WorkAdvance 
group respondents did not have consistent records across the data sources, 
and the results suggest that these respondents had challenges recalling when 
they participated in targeted sector training. 

Although the abovementioned data quality checks suggest that the providers’ data sys- 
tems contained a reliable record of program participation among WorkAdvance group mem- 
bers, this data source is subject to limitations. First, the estimated rates of participation in 
employment and training services or activities are more representative of participants who were 
actively engaged in WorkAdvance services than of those who were not engaged. For instance, 
employment service data for participants who had frequent spells of disengagement are likely 


10 For an example, see Table 4.1 in Tessler et al. (2014). 
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The WorkAdvance Study 
Appendix Table A.3 


Comparison of MIS and Survey Reports of Participation in 
Targeted Sector Training Since Random Assignment 
Among WorkAdvance Group Survey Respondents, by Site 



Per Scholas 

St. Nicks Alliance 

Madison 
Strategies Group 

Towards 

Employment 

Unadjusted outcome 

N 

% 

N 

% 

N 

% 

N 

% 

Not in training 

11 

3.8 

25 

12.3 

85 

28.6 

77 

27.0 

MIS-reported 









training only 

77 

26.8 

58 

28.3 

48 

16.2 

35 

12.2 

Survey-reported 









training only 

0 

0.0 

12 

5.9 

22 

7.4 

14 

4.9 

MIS and survey-reported 









training 

199 

69.3 

109 

53.2 

142 

47.8 

159 

55.6 

Sample size 

287 


205 


297 


286 



SOURCES: MDRC calculations from program tracking data managed by Per Scholas, St. Nicks Alliance, Madison 
Strategies Group, and Towards Employment, and from responses to the WorkAdvance Year 2 Survey. 

NOTES: N = number (sample size); MIS = management information system. 

Sample sizes may vary due to missing values. 


to be less complete than data for participants who remained engaged throughout the service 
period, since it is not always easy for providers to verily participants’ employment status 
without first establishing direct contact with those participants. 

Second, some providers do not systematically track certain activities in their MIS sys- 
tems, so the research team had to create customized data collection instruments (as an example, 
MDRC designed a career advancement coaching form and asked the providers to complete the 
forms for a three-month period during coaching sessions with WorkAdvance participants), and 
staff members within the same program may record information differently. As such, engage- 
ment in some services was not tracked for the full service period (for example, advancement 
coaching). In other cases, MDRC made reasonable adjustments to data that were collected 
throughout the service period and were sometimes inconsistent or incomplete. (For example, 
most providers were not able to report the date that participants dropped out of training, so 
MDRC used the midpoint of the training cycle as the participant’s last day in training.) Where 
this was not possible, MDRC relied on other data sources to help measure participation in 
WorkAdvance services (for example, the Year 2 Survey). 
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Summary participation outcome measures for employment services, education, and 
training were calculated for all study participants, as well as by site. Measures at 18 months are 
shown in the main report, and site-level definitions for measures are shown in Appendix Table 
C.3. 


Tests for statistical differences across sites were not perfonned. 


Year 2 Survey 

This section answers the following questions about the Year 2 Survey: (1) How many sample 
members responded to the survey? (2) How were these response rates achieved? (3) What 
procedures were followed to help ensure data quality? and (4) To what extent are survey 
respondents representative of the full sample? 

How Many Sample Members Responded to the Survey? 

MDRC contracted with Decision In formation Resources (DIR) to design and adminis- 
ter the Year 2 Survey. MDRC has worked with DIR on surveys for other studies, including 
evaluations of career advancement initiatives for low-income individuals and households. DIR 
staff members used their Computer- Assisted Telephone Interviewing (CATI) call center and 
database system to conduct all interviews. Per agreement with MDRC, DIR sought to interview 
at least 80 percent of both WorkAdvance group members and control group members at each of 
the four sites. As illustrated in Appendix Table A.4, DIR met this goal for the WorkAdvance 
groups and came close for the control groups at each WorkAdvance site. 

How Were These Response Rates Achieved? 

DIR put in place multiple efforts to achieve high survey response rates: 

Respondent location efforts. DIR used standard locator databases to find additional 
contact infonnation, beyond what was collected at the baseline interview. 

Marketing efforts. MDRC worked with members of the DIR survey team to create 
marketing materials (including letters, flyers, and e-mail messages) and scripts for marketing 
phone calls to encourage participants to complete survey interviews. 

Financial incentives. Respondents received a gift card after completing an interview. 
Notice of the gift cards was included in marketing materials. Throughout the fielding period, 
MDRC and DIR team members monitored survey response rates and at strategic points in time 
(for example, when response rates had dropped for several weeks in a row) increased the value 
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The WorkAdvance Study 
Appendix Table A.4 
Year 2 Survey Response Rates, by Site 



Per Scholas 

St. Nicks Alliance 

Madison 
Strategies Group 

Towards 

Employment 

WA 

C 

Total 

WA 

C 

Total 

WA 

C 

Total 

WA 

C 

Total 

Fielded sample (#) 

349 

341 

690 

242 

237 

479 

353 

344 

697 

349 

349 

698 

Respondent 













sample (#) 

287 

265 

552 

205 

179 

384 

297 

263 

560 

286 

276 

562 

Response rate (%) 

82.2 

77.7 

80.0 

84.7 

75.5 

80.2 

84.1 

76.5 

80.3 

81.9 

79.1 

80.5 


SOURCES: MDRC calculations from WorkAdvance baseline information form and responses to the Year 2 
Survey. 

NOTE: WA = WorkAdvance (program) group; C = control group. 


of the gift card. About 48 percent of respondents received a Visa gift card for $25 or $40, 35 
percent received either a $50 or $60 gift card, and 17 percent received a $100 gift card. In 
addition, unconditional $15 gift cards were included with the abovementioned marketing 
materials for approximately 40 percent of the fielding sample in an effort to help improve 
response rates. 

Field locators. DIR employed and trained a group of field locators and assigned them 
to personally contact study participants who had not yet completed an interview. Field locators 
set up appointments for interviews with DIR’s call center but did not interview respondents on 
site. DIR monitored the success rates of each field locator weekly. In some instances, locators 
were reassigned to different WorkAdvance locations to help boost response rates. DIR also 
replaced locators who were not successful in contacting study participants. 

Monitoring responses. During the fielding period, DIR prepared and shared with 
MDRC weekly reports on survey response rates, organized by WorkAdvance site and research 
group. Members of the DIR and MDRC teams reviewed these reports weekly and made 
adjustments to fielding efforts within WorkAdvance sites (for example, allocation of field 
locators) in response to identification of low response rates or relatively large differences in 
response rates by research group. 
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What Procedures Were Followed to Help Ensure Data Quality? 

Survey design. Most questions included in the survey had been used in previous sur- 
veys on employment and training, including surveys administered by DIR for previous MDRC 
studies. The survey questiomiaire was reviewed by a team that included experts in survey 
design and researchers in training initiatives. 

CATI system testing. Before fielding the survey, members of MDRC’s research team 
and corporate Survey Unit and the DIR survey team tested the programming of DIR’ s CATI 
system. Team members completed several rounds of review of technical issues in the CATI 
system and modified the questionnaire and CATI system accordingly. 

Monitoring interviewer training. Before fielding the survey, members of MDRC’s 
research team and corporate Survey Unit reviewed DIR’s interviewer training materials and 
scripts and attended a training session for interviewers. 

Identifying survey respondents. MDRC shared with DIR respondents’ dates of birth 
and the last four digits of their Social Security numbers. At the start of each interview, respond- 
ents were asked to provide this infonnation to the interviewer to verily their identity. No proxy 
interviews by third parties were allowed. 

Interviewing respondents. All interviews were conducted by phone with interviewers 
at DIR’s call center. Field locators did not interview respondents. 

DIR review of survey responses. DIR recorded all survey interviews and made the re- 
cordings available for review. For a sample of respondents, DIR supervisors monitored inter- 
views in real time. In addition, DIR conducted verification calls on 1 0 percent of each CATI 
interviewer’s completed interviews and on 10 percent of each field locator’s completed contacts 
that led to an interview. Finally, for a sample of respondents, DIR supervisors reviewed every 
recorded response and compared the response with the value recorded in the CATI system. 1 1 

MDRC review of survey responses. During the first months of survey fielding, DIR 
transmitted to MDRC two test files of survey responses. Members of MDRC’s research team 
processed the data in these test files and ran quality control checks on the data. Team members 
also checked item response rates, skip patterns, missing responses, outlier responses, and 
responses that appeared to be inconsistent with responses to related questions, with the audio 


"Due to anomalous data values that were discovered during their routine quality checks, the survey firm 
launched an investigation to review all attempted and completed interviews that were conducted within the 
timeframe that these anomalous values were generated. They verified that records for three participants were 
affected and dropped these records from survey data files that were submitted to MDRC for processing and 
analysis. 
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recording, or both. MDRC performed quality control checks on individual items and composite 
measures (for example, total weekly earnings at interview). MDRC shared examples of problem 
responses with DIR. DIR team members then reviewed each issue and, where necessary, 
reviewed the recorded verbal response to determine whether the interviewer had entered the 
response incorrectly in DIR’s CATI system. MDRC and DIR repeated these procedures 
following secure transmission to MDRC of each survey file. DIR corrected responses as needed 
based on these reviews. 

To What Extent Are Survey Respondents Representative of the 

Full Sample? 

The WorkAdvance Year 2 Survey provided data on research sample members’ partici- 
pation in services, employment, and various measures of overall well-being. Survey interviews 
were conducted between 18 and 30 months after each individual entered the study. The survey 
nonresponse analysis presented in this appendix examines whether the survey-based outcomes 
presented in the main report (which cover only Year 2 Survey respondents) can be generalized 
to the full research sample. Findings are presented for each site individually. Five main analyses 
were perfonned: 

Comparison of respondents and nonrespondents. Because the full WorkAdvance 
sample was contacted for the survey, respondents are expected to have characteristics similar to 
nonrespondents. 12 A logistic regression was run to test whether key baseline characteristics 
could predict whether an individual was a respondent to the Year 2 Survey at each site. The 
models included several characteristics that were regressed on a dichotomous survey response 
indicator (1 = survey respondent; 0 = nonrespondent). 13 If this model is statistically significant it 
indicates that survey respondents have different background characteristics from nonrespond- 
ents. This would not imply that the causal effects of WorkAdvance are biased. It would simply 
imply that those causal effects are measured for a slightly different sample than was used for the 
administrative records analysis. 

Comparison of WorkAdvance and control group respondents. Because of random 
assigmnent, WorkAdvance group members are expected to be similar to control group mem- 
bers, and thus respondents should also be similar across research groups. A logistic regression 
analysis was conducted to test whether key baseline characteristics could predict whether a 


12 For the purpose of the nonresponse analyses, nonrespondents are defined as sample members who did 
not respond to the survey between 18 and 30 months of study entry. This includes sample members who did 
not respond at all to the Year 2 Survey. 

l3 The baseline characteristics used in the logistic regression models differed somewhat by site. The full list 
of baseline characteristics for each site can be found in the tables. See Appendix Table A.5 for a list ofbaseline 
characteristics that were used in each site’s regression model. 
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respondent to the Year 2 Survey was in the WorkAdvance group. The model included several 
characteristics that were regressed on a dichotomous WorkAdvance group indicator (1 = 
WorkAdvance group; 0 = control group). If this model is statistically significant the implica- 
tions are potentially more serious. It would imply that differential attrition, due to the survey 
response process, has (to some extent) undermined the comparability of the research groups and 
thereby the internal validity of the experiment. Such results would require further exploration to 
measure how large any differences between the groups are and how correlated those character- 
istics are with the key outcomes of the experiment. 

Comparison of employment and earnings outcomes, through the first half of Year 
2, between the research sample and the respondent sample. Individuals across the research 
and respondent samples are expected to show similar levels of and impacts on Ul-covered 
employment and earnings. This test examines whether that is the case. 

Classification tree analysis. In order to assess for nonresponse bias more deeply, a 
classification and decision tree analysis (a form of data mining) was used to check whether any 
specific group or groups of study participants were more or less likely to have responded to the 
Year 2 Survey. 

Multiple imputation. Missing data can be especially troublesome and can lead to bias 
when reporting and interpreting outcomes. Multiple imputation was used as a final check of the 
implications of survey nonresponse on the validity of the survey results. This approach, recom- 
mended as a best practice by the Institute of Education Sciences, 14 involves simulating what the 
survey results would have looked like had the survey response rate been 100 percent. Multiple 
imputation was used as a sensitivity check of the Year 2 Survey results. Missing values were 
imputed for a range of survey-based measures to check for bias due to both survey and item 
nonresponse. The main check here determines whether the findings from the imputed survey 
results look similar to the pattern of findings from the actual survey results. If so, this would 
provide strong evidence of the validity of the survey findings. 

Below is a summary of findings from the five-part analysis for each of the four Work- 
Advance sites. Overall, no serious nonresponse bias issues were found among the respondent 
sample, so no adjustments for nonresponse were imposed. 

Per Scholas 

Overall, there is no serious nonresponse bias among the sample enrolled at Per Scholas. 


4 Puma. Olsen, Bell, and Price (2009). 
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• Appendix Table A.5 shows that survey respondents at Per Scholas are simi- 
lar to nonrespondents across the measured baseline characteristics; the lo- 
gistic regression model is not statistically significant (p-value = 0.194). Nev- 
ertheless, the results do show that Per Scholas respondents are more likely 
than nonrespondents to be black, less likely to have received UI benefits be- 
fore study entry, and especially more likely to have been randomly assigned 
after 2011. 

• The statistically insignificant results (p-value = 0. 146) of the logistic regres- 
sion model in Appendix Table A.6 show that WorkAdvance group respond- 
ents at Per Scholas are similar, on average, to control group respondents. 

• Appendix Table A.7 shows that the research group means and impacts on UI 
employment and earnings measures for the research and respondent samples 
at Per Scholas are fairly consistent. For two of three measures, the impacts 
are somewhat larger for the respondent sample compared with the research 
sample, but the estimated impacts are statistically insignificant for both the 
research and respondent samples on all three measures. Using the “ever em- 
ployed” outcome as an example, there is an impact of approximately 4 per- 
centage points among the research sample and an impact of approximately 4 
percentage points among the respondent sample. Note these impacts are sta- 
tistically insignificant for both samples. 

• The classification tree analysis did not reveal any noteworthy groups of Per 
Scholas sample members who were more or less likely to respond to the 
Year 2 Survey (Appendix Figure A.1). This is consistent with the abovemen- 
tioned logit model results. 

• In Appendix Table A. 8, multiple imputation was used to assign values to 
measures with missing values due to both survey and item nonresponse. The 
multiply imputed impacts are consistent with the impacts shown in the main 
report for Per Scholas. For example, Table 5.1 shows a statistically signifi- 
cant 10 percentage point impact on current employment at this site; in Ap- 
pendix Table A. 8, the multiply imputed 13 percentage point impact on this 
outcome is also statistically significant. 

St. Nicks Alliance 

Overall, while there are some differences between respondents and nonrespondents, 
there is no serious nonresponse bias among the sample enrolled at St. Nicks Alliance. 
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The WorkAdvance Study 
Appendix Table A.5 


Estimated Regression Coefficients for the Probability of Being a Respondent to the WorkAdvance Year 2 Survey, by Site 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Characteristic 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

WorkAdvance group 

0.258 

0.753 *** 

0.494 ** 

0.242 

Female 

-0.287 

0.488 

-0.180 

0.545 * 

Black 

0.431 * 

0.871 ** 

0.186 

-0.208 

Hispanic 

0.330 

0.304 

0.052 

- 0.688 

Age 

-0.008 

-0.006 

0.009 

- 0.012 

Under age 24 

0.019 

0.406 

0.386 

-0.497 

Number of children living with respondent 

0.032 

0.142 

0.171 * 

0.005 

Born in state 

- 0.010 

1.385 *** 

0.398 * 

-0.768 ** 

Highest level of education 

-0.008 

NA 

NA 

NA 

Highest level of education is GED or less 

NA 

-0.167 

NA 

NA 

Highest level of education is 2- or 4-year degree or more 

NA 

NA 

0.907 ** 

NA 

Highest level of education is high school, GED, or less 

NA 

NA 

NA 

-0.208 

Already has a license/certificate in targeted sector 

0.023 

-0.833 

-0.710 ** 

0.107 

Currently employed at baseline 

-0.272 

0.362 

-0.195 

0.565 * 

Monthly family income 

0.000 

0.000 

0.000 

0.000 

Employed in at least 1 quarter in year prior to RA 

-0.338 

0.230 

0.169 

-0.318 

Employed in all 4 quarters in year prior to RA 

0.070 

1.068 * 

0.501 

-0.791 ** 

Number of quarters employed in 3 years prior to RA 

0.082 

- 0.021 

- 0.012 

0.044 

Total earnings in quarter prior to RA 

0.000 

0.000 

0.000 

0.000 

Total earnings in 3 years prior to RA 

Received unemployment insurance benefits in 

0.000 

0.000 

0.000 

0.000 

3 years prior to RA 

-0.464 * 

0.780 ** 

0.249 

NA 

Randomly assigned to environmental remediation sector 

NA 

-0.260 

NA 

NA 

Randomly assigned to health care sector 

NA 

NA 

NA 

0.155 

Randomly assigned in 201 1 

-0.763 *** 

-1 445 *** 

-0.915 *** 

0.182 

Randomly assigned in 2012 

-0.267 

-0.457 

-0.104 

0.489 ** 

R-squared 

0.041 

0.152 

0.088 

0.058 

Chi-square statistic 

28.602 

78.827 

63.859 

41.908 

P-value of chi-square statistic 

0.194 

0.000 *** 

0.000 *** 

0.009 *** 

Sample size (total = 2,564) 

690 

479 

697 

698 


(continued) 






Appendix Table A.5 (continued) 


SOURCE: MDRC calculations from the WorkAdvance baseline information form and unemployment insurance administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: NA = not applicable; GED = General Education Development certificate; RA = random assignment. 

A chi-square test was applied to differences between outcomes for the early and late cohorts in the analysis for the full research sample. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The WorkAdvance Study 
Appendix Table A.6 


Estimated Regression Coefficients for the Probability of Being a WorkAdvance 
Group Respondent to the WorkAdvance Year 2 Survey, by Site 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Characteristic 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

Parameter Estimate 

Female 

-0.239 

0.126 

0.371 

-0.057 

Black 

0.218 

-0.300 

-0.013 

0.403 * 

Hispanic 

0.157 

-0.417 

-0.757 * 

-0.112 

Age 

-0.017 

0.029 ** 

-0.003 

-0.003 

Under age 24 

0.283 

0.483 

0.288 

-0.160 

Number of children living with respondent 

-0.002 

-0.119 

0.104 

0.060 

Born in state 

0.089 

-0.175 

-0.235 

0.054 

Highest level of education' 1 

0.097 

0.119 

-0.095 

0.175 

Already has a license/certificate in targeted industry 

-0.294 

0.917 

0.206 

0.480 ** 

Currently employed at baseline 

-0.661 ** 

-0.080 

-0.078 

-0.232 

Monthly family income 

0.000 

0.000 

0.000 

0.000 

Employed in at least 1 quarter in year prior to RA 

-0.175 

0.146 

0.274 

-0.066 

Employed in all 4 quarters in year prior to RA 

0.027 

0.001 

0.330 

0.559 * 

Number of quarters employed in 3 years prior to RA 

-0.025 

0.003 

-0.036 

-0.061 

Total earnings in quarter prior to RA 

0.000 

0.000 

0.000 

0.000 

Total earnings in 3 years prior to RA 

Received unemployment insurance benefits in 

0.000 

0.000 

0.000 

0.000 * 

3 years prior to RA 

0.508 ** 

-0.332 

0.335 * 

NA 

Randomly assigned in 201 1 

0.205 

0.594 

0.121 

-0.015 

Randomly assigned in 2012 

0.005 

-0.002 

-0.047 

0.264 

Randomly assigned to environmental remediation sector 

NA 

-0.056 

NA 

NA 

Randomly assigned to health care sector 

NA 

NA 

NA 

-0.151 

R-squared 

0.049 

0.047 

0.043 

0.046 

Chi-square statistic 

27.813 

18.374 

24.525 

26.738 

P-value of chi-square statistic 

0.146 

0.826 

0.489 

0.221 

Sample size 

552 

384 

560 

562 
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Appendix Table A.6 (continued) 


SOURCES: MDRC calculations from the WorkAdvance baseline information form and from 
unemployment insurance wage records from the New York State Department of Labor, Ohio 
Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: RA = random assignment; NA = not applicable. 

A chi-square test was applied to differences between outcomes for the WorkAdvance and control 
groups in the analysis for the full research sample. Statistical significance levels are indicated as follows: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

a Highest level of education measures used in the regression model varied by site. 


• Appendix Table A.5 shows that survey respondents at St. Nicks Alliance dif- 
fer from nonrespondents across the measured baseline characteristics; the lo- 
gistic regression model is statistically significant (p-value < 0.001). One of 
the key drivers of the differences between respondents and nonrespondents is 
that respondents were more likely to be WorkAdvance group members and 
less likely to have been randomly assigned in 2011. This is consistent with 
DIR’s early challenges in completing survey interviews with control group 
members for this site. 

• The average WorkAdvance group respondent at St. Nicks Alliance is not dif- 
ferent from the average control group respondent (shown in Appendix Table 
A.6). Overall, the logistic regression model is not statistically significant (p- 
value = 0.826). 

• Appendix Table A.7 shows that the research group means and impacts on UI 
employment and earnings measures for the research and respondent samples 
at St. Nicks Alliance are fairly consistent. The impacts are somewhat larger 
for the research sample compared with the respondent sample, but none of 
the estimated impacts are statistically significant for either sample. Using the 
“ever employed” outcome as an example, there is a 5 percentage point im- 
pact among the research sample but only a 1 percentage point impact among 
the respondent sample, but these impacts are statistically insignificant for 
both samples. 

• The classification tree analysis confinned that at St. Nicks Alliance, control 
group members who were randomly assigned in 20 1 1 were less likely to re- 
spond to the survey (Appendix Figure A.l). This is consistent with the very 
large program-control group differentials for the first few cohorts of survey 

respondents. 
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The WorkAdvance Study 
Appendix Table A.7 


Quarters 2 to 7 Impacts on Employment and Earnings for the Full and 
Survey Respondent Samples, by Site 



WorkAdvance Group 

Control Group 

Difference 

Percentage 


Outcomes 

Average 

N 

Average 

N 

(Impact) 

Difference 

P-Value 

Per Scholas 








Ever employed (%) 

Research sample 

83.9 

349 

80.4 

341 

3.5 

4.3 

0.221 

Respondent sample 

85.3 

288 

81.0 

266 

4.3 

5.2 

0.170 

Average quarterly employment (%) 
Research sample 

57.8 

349 

54.6 

341 

3.2 

5.8 

0.235 

Respondent sample 

59.1 

288 

56.2 

266 

2.9 

5.2 

0.314 

Total earnings ($) 

Research sample 

17,144 

349 

15,595 

341 

1,549 

9.9 

0.214 

Respondent sample 

17,919 

288 

15,983 

266 

1,936 

12.1 

0.171 

St. Nicks Alliance 








Ever employed (%) 

Research sample 

83.7 

242 

78.7 

237 

5.0 

6.3 

0.158 

Respondent sample 

85.0 

205 

83.7 

179 

1.3 

1.6 

0.717 

Average quarterly employment (%) 
Research sample 

55.2 

242 

53.0 

237 

2.1 

4.0 

0.502 

Respondent sample 

58.0 

205 

56.5 

179 

1.5 

2.7 

0.655 

Total earnings ($) 

Research sample 

16,348 

242 

16,437 

237 

-89 

-0.5 

0.955 

Respondent sample 

17,543 

205 

17,621 

179 

-79 

-0.4 

0.966 


Madison Strategies Group 

Ever employed (%) 


Research sample 

91.4 

353 

89.4 

344 

2.0 

2.3 

0.355 

Respondent sample 

93.4 

297 

90.4 

263 

3.0 

3.3 

0.185 

Average quarterly employment (%) 

Research sample 

71.6 

353 

69.2 

344 

2.4 

3.5 

0.325 

Respondent sample 

74.6 

297 

71.8 

263 

2.8 

3.9 

0.285 

Total earnings ($) 

Research sample 

21,602 

353 

20,461 

344 

1,141 

5.6 

0.364 

Respondent sample 

23,467 

297 

21,301 

263 

2,166 

10.2 

0.122 


(continued) 
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Appendix Table A.7 (continued) 


Outcomes 


WorkAdvance Group Control Group Difference 
Average N Average N (Impact) 


Percentage 
Difference P-Value 


Towards Employment 

Ever employed (%) 


Research sample 

87.6 

349 

83.1 

349 

4.5 * 

5.4 

0.080 

Respondent sample 

87.0 

287 

81.6 

280 

5.4 * 

6.6 

0.066 

Average quarterly employment (%) 

Research sample 

63.4 

349 

62.0 

349 

1.4 

2.2 

0.572 

Respondent sample 

63.3 

287 

60.2 

280 

3.1 

5.2 

0.251 

Total earnings ($) 

Research sample 

15,867 

349 

15,153 

349 

714 

4.7 

0.452 

Respondent sample 

16,010 

287 

14,106 

280 

1,903 * 

13.5 

0.068 


SOURCES: MDRC calculations from the WorkAdvance baseline information form and from unemployment 
insurance administrative records provided by New York State Department of Labor, Ohio Department of Jobs and 
Family Services, and Oklahoma Employment Security Commission. 

NOTES: N = number (sample size). 

A two-tailed t-test was applied to the difference between outcomes for the program and control groups in the 
analysis for the full research sample. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

• The multiply imputed impacts in Appendix Table A. 8 are consistent with the 
impacts shown in the main report for St. Nicks Alliance. For example, Table 
5.1 shows an insignificant, negative 1 percentage point impact on ever being 
employed at this site; in Table A. 8, the multiply imputed negative 2 percent- 
age point impact on this outcome is also statistically insignificant. 

Madison Strategies Group 

Overall, while there are some differences between respondents and nonrespondents, 
there is no serious nonresponse bias among the sample enrolled at Madison Strategies Group. 

• Appendix Table A.5 shows that survey respondents at Madison Strategies 
Group differ from nonrespondents across the measured baseline characteris- 
tics; the logistic regression model is statistically significant (p-value < 0.001). 

Here, too, one of the key drivers of the differences between respondents and 
nonrespondent is the study entry year, with responses being less likely 
among those randomly assigned in 2011. 
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The WorkAdvance Study 
Appendix Figure A.l 

Classification and Regression Tree (CART) Results, by Site 


Per Scholas St. Nicks Alliance 


Node ID: 

1 

0: 

20.04% 

1: 

79.96% 

Count: 

479 


f 


Node ID: 

1 

0: 

19.71% 

1: 

80.29% 

Count: 

690 


1/0 RAd in 2011 


Madison Strategies Group 



SOURCES: MDRC calculations from the WorkAdvance baseline information form, survey administration data from 
Decision Information Resources, and unemployment insurance administrative records from New York State Department 
of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: RAd = randomly assigned; BIF = baseline information form. 

Percentages in the “1” category represent all survey respondents, including a few sample members who 
responded outside the 18- to 30-month follow-up window that was used for the Year 2 Survey analysis. As such, 
these percentages may differ from response rates given in Appendix Table A.4. 

The CART technique was used to predict the probability of survey interview response by classifying sample 
members based on their employment and wage history, their baseline characteristics, the number of attempts made 
by the survey firm to reach the sample member for the interview, and the mode in which the interview was 
completed. This classification process is performed separately for each site. 

The gray shading of boxes denotes a greater proportion of survey respondents in the indicated group. The darker 
the shading, the larger the proportion of survey respondents in that group. 

For Per Scholas, Madison Strategies Group, and Towards Employment, sample members’ baseline and survey- 
related characteristics did not predict likelihood of survey response. This is depicted in the above illustration by no 
“tree” growth. At St. Nicks Alliance, two characteristics were identified as predictive of survey response: year of 
random assignment and research group. The tree growth for this site is interpreted as follows: (1) sample members 
randomly assigned after 2011 were more likely to respond to the survey than those randomly assigned in 201 1, and 
(2) program group members who were randomly assigned in 201 1 were more likely to respond to the survey than 
control group members who were randomly assigned in the same year. 
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The WorkAdvance Study 
Appendix Table A.8 

Multiply Imputed Year 2 Impacts on Employment, Earnings, and Income, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 


Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Ever employed (%) 

94.5 

88.5 

6.0 *** 

89.1 

90.6 

-1.5 

95.7 

92.7 

3.0 

89.7 

88.1 

1.6 

Currently employed 

74.8 

62.0 

12 7 *** 

64.7 

67.1 

-2.3 

75.5 

70.6 

5.0 

65.2 

61.8 

3.5 

Currently employed 

in targeted sector (%) 

46.9 

15.3 

31.6 *** 

23.4 

14.2 

9.2 * 

54.7 

37.4 

17 3 *** 

37.4 

24.0 

13 4 *** 

Average number of hours worked per 

week at current or most recent job 

34.1 

32.5 

1.6 

36.2 

34.5 

1.7 

41.8 

39.6 

2.1 * 

33.2 

31.6 

1.5 

Not employed since RA (%) 

4.6 

8.9 

-4.4 * 

9.3 

7.1 

2.2 

3.4 

4.4 

- 1.1 

9.3 

10.6 

-1.3 

Employed part-time 

31.7 

34.5 

- 2.8 

22.7 

30.8 

- 8.1 * 

17.3 

22.7 

-5.3 * 

32.7 

38.0 

-5.3 

Employed full-time ' 1 

63.7 

56.5 

7.2 

68.0 

62.1 

5.9 

79.3 

72.9 

6.4 * 

58.0 

51.4 

6.6 

Average hourly wage at current or 

most recent job ($) 

15.28 

12.84 

2 44 *** 

13.13 

13.16 

-0.03 

12.48 

11.81 

0.66 

10.18 

10.29 

- 0.11 

Percentage of months employed 

56.9 

50.7 

6.2 ** 

51.7 

51.2 

0.5 

65.2 

65.7 

-0.5 

52.9 

58.4 

-5.5 * 

Offered opportunities for career 

advancement (%) 

67.3 

49.1 

lg 1 *** 

59.0 

58.1 

0.9 

64.7 

55.7 

0 9** 

58.2 

46.9 

j l 3 *** 

Household income in prior month ($) 

2,129 

1,650 

47 g *** 

2,018 

2,319 

-301 

2,053 

2,106 

-53 

1,446 

1,557 

-111 

Job earnings (%) 

79.8 

65.7 

^4 2 *** 

69.1 

72.5 

-3.4 

79.6 

76.8 

2.8 

71.0 

64.1 

6.9 

Unemployment insurance (%) 

4.1 

7.9 

-3.8 

6.9 

3.3 

3.6 

2.4 

2.5 

- 0.1 

3.5 

3.9 

-0.4 

Sample size (total = 2,564) 

349 

341 


242 

237 


353 

344 


349 

349 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment. 

Multiple imputation was used to assign values to measures with missing values due to both survey and item nonresponse. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

a Full-time employment is defined as working 35 or more hours in a week. 



• The statistically insignificant results (p-value = 0.489) of the logistic regres- 
sion model in Appendix Table A.6 show that WorkAdvance group respond- 
ents at Madison Strategies Group are similar, on average, to control group re- 
spondents. 

• Appendix Table A.7 shows that the research group means and impacts on UI 
employment and earnings measures for the research and respondent samples 
at Madison Strategies Group are fairly consistent. The earnings impacts are 
somewhat larger for the respondent sample (at $2, 1 66) compared with the re- 
search sample (at $1,141), but these estimated differences are not statistically 
significant for either sample. 

• The classification tree analysis did not reveal any noteworthy groups of Mad- 
ison Strategies Group sample members who were more or less likely to re- 
spond to the Year 2 Survey (Appendix Figure A.l). This is consistent with 
the results above. 

• The multiply imputed impacts in Appendix Table A. 8 are consistent with the 
impacts shown in the main report for Madison Strategies Group. For exam- 
ple, Table 5.1 shows an insignificant 2 percentage point impact on ever being 
employed at this site among the respondent sample; this is similar to the mul- 
tiply imputed and insignificant 3 percentage point impact shown in Table A.8 
for the research sample. 

Towards Employment 

• Appendix Table A.5 shows that survey respondents at Towards Employment 
differ from nonrespondents across the measured baseline characteristics; the 
logistic regression model is statistically significant (p-value = 0.009). Some 
of the key drivers of the differences between respondents and nonrespond- 
ents include gender, birth state, employment history and status at study entry, 
and year of study entry. 

• The average WorkAdvance group respondent at Towards Employment is not 
different from the average control group respondent (shown in Appendix Ta- 
ble A.6). Overall, the logistic regression model is not statistically significant 
(p-value = 0.221). 

• Appendix Table A.7 shows that the research group means and impacts on UI 
employment and earnings measures for the research and respondent samples 
at Towards Employment are fairly consistent. Still, the earnings impacts (at 
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$1,903) are somewhat elevated and statistically significant for the respondent 
sample compared with the research sample (at $714), suggesting some bias 
due to survey nonresponse. 

• The classification tree analysis did not reveal any noteworthy groups of To- 
wards Employment sample members who were more or less likely to re- 
spond to the Year 2 Survey (Appendix Figure A.l). This is consistent with 
the abovementioned logit model results. 

• The multiply imputed impacts in Appendix Table A. 8 are consistent with the 
impacts shown in the main report for Towards Employment. For example, 
Table 5.1 shows an insignificant 2 percentage point impact on ever being 
employed at this site among the respondent sample; this is similar to the mul- 
tiply imputed and insignificant 2 percentage point impact shown in Table A.8 
for the research sample. 


Cost Analysis Data and Findings 

Selection of Year 2 as the Steady-State Period 

As stated in Chapter 4, Year 2 of the WorkAdvance program was selected as the 
steady-state period for the cost analysis. Year 3 might appear at first to be a better choice, being 
even less subject to the inclusion of start-up costs than Year 2; moreover, by Year 3 postem- 
ployment services had been given more opportunity to mature. But random assignment ended 
for WorkAdvance at the end of June 2013, only three months after Year 3 had begun. While 
new participants were enrolled into the program after random assignment ended, there was a 
gap of four to eight months (depending on the site) between the end of random assignment and 
the start of new enrollment into WorkAdvance. 15 As a consequence, Year 3 is not sufficiently 
representative of a steady state or even of normal program operations. Thus, in reporting the 
cost findings, costs for Year 2 are given greater prominence than those for Year 3. 

Still, Year 2 falls short of accurately reflecting the steady state because the entry of new 
participants into WorkAdvance during Year 1 was slower than program entry during Year 2 — 
considerably slower in some sites. Moreover, the entry period in Year 1 varied by site and did 
not begin until well after April 2011 at any of them. As a result, in Year 2 there were fewer 


15 While all sites suspended enrollment into WorkAdvance during this gap, they continued to run other 
programs. Per Scholas was unique among the sites in that it continued to enroll new participants for training 
and services resembling WorkAdvance. However, those activities did not fall under the umbrella of WorkAd- 
vance. 
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persons far enough along in their two-year period of eligibility to have graduated from occupa- 
tional skills training than there would have been in a true steady state, and, as a consequence, 
the Year 2 cost of services received after graduating is probably understated relative to costs 
incurred earlier. (During Year 3, in contrast, because of the long break in enrollment there were 
fewer participants who had not yet graduated from occupational skills training than there would 
have been in the steady state, so pregraduation costs incurred that year are probably slightly 
understated relative to costs incurred after graduation.) Based on an assessment of costs tied to 
postemployment services at different stages of the program, MDRC estimates that gross costs in 
Year 2 were probably understated on the order of 1 percent to 2 percent at the different sites. 
The sensitivity of the cost findings to Year 2 conditions was examined, and postemployment 
costs from Year 3 are substituted for those of Year 2 to account for the Year 2 understatement 
of such costs. 

Sources of Cost Data 

Financial reports detailing expenditures, when submitted to the Mayor’s Fund to Ad- 
vance New York City, included a certification statement and were substantiated through 
additional fiscal oversight. They were submitted using an online grants management system 
(GMS) that included the budgets for each program year, and expenditures in each report were 
listed as total amounts spent against a line item during the designated reporting period. Reported 
information was exported from the GMS to Excel for further analysis. 

For purposes of determining net costs, specific survey questions were selected as key 
indicators of the receipt of preemployment services and support, occupational skills training, 
and postemployment retention and advancement services by control group members. These 
questions were used to isolate costs that would have existed in the absence of WorkAdvance. 
Costs associated with the activities mentioned above were established for each of the three 
relevant cities based on financial data collected and made public by the U.S. Department of 
Education 16 and the U.S. Department of Labor, 17 and using estimates developed by MDRC for 
services not captured in the available data. 


16 The National Center for Education Statistics, housed within the Institute of Education Sciences, publish- 
es core expenses per full-time equivalent enrollment at public and private for-profit and not-for-profit educa- 
tional institutions, derived from data that each institution is required to report to the Department of Education 
as part of their participation in federal student financial aid programs. Expense data for fiscal year 2013 was 
used, drawn from the Integrated Postsecondary Education Data System, which is available online. 

17 Employment and Training Administration Employment Service (Wagner-Peyser) Program Year 2013 
final allotments are available online. 
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Variation in the Gross Cost Estimates 

The two New York providers, St. Nicks Alliance and Per Scholas, are at the higher end 
of the narrow range of total per participant costs in Year 2, with Madison Strategies Group and 
Towards Employment at the lower end. There are at least three possible explanations for this. 
The first is simply that salaries and other operating costs tend to be higher in New York City 
than in Tulsa or Cleveland. A second potential explanation is tied to the dual track operated 
initially by Madison Strategies Group and Towards Employment, in which, as discussed in the 
main text of the report, some WorkAdvance enrollees were placed directly into jobs while 
others received the full suite of training and supportive services. While the percentage of 
individuals at each of the two dual-track sites given training from the start was significantly 
larger than the share directed into jobs, fewer resources were expended on those placed directly 
into jobs, and the difference is substantial enough to have an effect on costs. Consequently, 
when aggregate costs are divided by the number of WorkAdvance participants at each site, the 
larger numbers of individuals receiving less costly service at Madison Strategies Group and 
Towards Employment may be bringing down the gross costs per participant at those two sites. 

A third possibility arises from the fact that St. Nicks Alliance and Per Scholas had few- 
er participants than Madison Strategies Group or Towards Employment. During Year 2, for 
example, St. Nicks Alliance had only 68 percent as many entrants as either Madison Strategies 
Group or Towards Employment, yet its aggregate gross costs in Year 2 were 81 percent as large 
as either of those two sites. Similarly, but less dramatically, Per Scholas had only about 80 
percent as many Year 2 entrants as Madison Strategies Group or Towards Employment, but had 
aggregate Year 2 gross costs that were 88 percent as large. Thus, the numbers of participants at 
St. Nicks Alliance and Per Scholas were smaller relative to aggregate costs than they were at 
Madison Strategies Group and Towards Employment, suggesting that economies of scale might 
have existed in the operation of WorkAdvance. For example, the number of participants 
assigned to each of the job developers and career coaches at St. Nicks Alliance and Per Scholas 
may have been smaller than at the other two sites even though each had the capacity to serve 
more. 


Unemployment Insurance Wage and Benefits Data 

Data Collection 

MDRC negotiated separate data sharing agreements with three state agencies (the New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma 
Employment Security Commission) to facilitate the collection of UI wage and benefits data to 
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measure WorkAdvance’s effect on employment and earnings. These data were collected every 
6 to 12 months. 18 

Each shipment included the most recent quarter of data available and at least five quar- 
ters of historical data. This allowed for an “overlap” across data files, and records were checked 
for any updated infonnation from one file to the next. 

Procedures used to ensure data quality include: 

• Checks for duplicate records across and within data files. 

o Duplicates by person, quarter, and employer: One record was dropped 
and the other record was updated to report the two records’ average earn- 
ings amount. 

o Duplicates by person, quarter, and earnings amount: If the two records 
were thought to be from the same employer (that is, the employer ID had 
been updated from one file to the next), the record from the earlier file 
was dropped. 

• Checks for outliers in quarterly earnings and benefits amounts. (More infor- 
mation on how outliers were handled can be found in the next section.) 

• Checks for incomplete quarters of data. 

• Checks of average quarterly earnings and benefits amounts and average quar- 
terly employment by calendar quarter and relative quarter. 

o Where data were available, checks of average quarterly earnings and av- 
erage quarterly employment within the targeted sector(s) were performed 
as well. 19 


Estimating Effects of WorkAdvance 

Before estimating the effects of the WorkAdvance programs, MDRC conducted an analysis 
planning process. All the analytic decisions discussed in this section — including sample 
definitions, outcome measure and subgroup definitions, statistical procedures, and other data 
analysis issues — were prespecified during that planning process. Prespecification is a key 


18 Because of the limited amount of historical UI benefits data available in Ohio, MDRC did not collect UI 
benefits data for Towards Employment sample members. 

19 Sector employment information was not available in New York UI records. 
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means of safeguarding a statistical study from drawing false conclusions. By prespecifying, 
analysts are limited in their ability to “search for impacts” when they may not be present. 

Power Analysis 

In the Subgrantee Evaluation Plan, MDRC indicated the analysis would be conducted at 
the sector level. Each WorkAdvance provider targeted only one sector, with the exception of 
Towards Employment, which targeted two sectors — health care and manufacturing. Roughly 
half that provider’s sample was randomly assigned within the health care program and half 
within the manufacturing program. In the original study design, Towards Employment part- 
nered with another provider in northeast Ohio (Compass), which also targeted the same two 
sectors. The original intention was to pool the samples within sectors across these two sites. 
However, because the Compass sample was dropped from the study, the sample sizes within the 
health care and manufacturing sectors were diminished. A power analysis performed by MDRC 
found these sample sizes to be too small to detect any impacts at a commonly accepted level. 20 
For this reason, a decision was made, before impact data were available, to switch from the 
sector to the site level of analysis. This decision affected only the Ohio site, because at the other 
three sites the sector is cotenninous with the site. 

Appendix Table A.9 shows the minimum detectable effects (or MDEs) calculated dur- 
ing the power analysis. MDEs are a key measure of statistical power. Conventionally, an MDE 
is the smallest true effect that has an 80 percent chance of being statistically significant at the 10 
percent level. MDEs are commonly expressed in effect size units (specifically, in temis of 
standard deviations) to pemiit comparisons across outcomes with different units. This expres- 
sion of an MDE is referred to as a minimum detectable effect size (or MDES). A common rule 
of thumb is to ensure studies have sufficient power to detect impacts at or below an MDES of 
0.2, which is a common threshold for a “small” effect size. 21 

As shown in the top half of Appendix Table A.9, the MDESs for the administrative 
records sample at all four providers are between 0.167 (at Towards Employment) and 0.204 (at 
St. Nicks Alliance). These are all below or close to the 0.2 threshold. Assuming 50 percent of 
the control group was employed (that is, the standard deviation is 0.5), 22 these MDESs translate 
into MDEs of between 8.4 and 10.2 for percentage measures. 


20 See Cohen (1992) and Hill, Bloom, Black, and Lipsey (2007). 

“'The 0.2 rule of thumb comes from Cohen. Cohen defined an effect size of 0.2 as “small,” 0.5 as “medi- 
um,” and 0.8 as “large.” Lipsey, another prominent researcher, sets the threshold lower. To Lipsey, an effect 
size of 0. 15 or lower is small. See Cohen (1992) and Hill, Bloom, Black, and Lipsey (2007). 

22 This assumption is the worst-case scenario. The point of maximum variance for a percentage measure is 
0.5 (a control group level of 50 percent). At that point, an MDES of 0.2 translates into an MDE of 10 percent- 

(continued) 
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The WorkAdvance Study 
Appendix Table A.9 
Minimum Detectable Effects, by Sample 


Sample 


MDE b 

Sample Employment (%) Earnings ($) 

Size R-squared a MDES SD = 0.4 SD = 0.5 SD = 2,000 SD = 3,500 


Administrative records sample 

Per Scholas 

690 

0.183 

0.171 

6.8 

8.6 

342 

599 

St. Nicks Alliance 

479 

0.193 

0.204 

8.2 

10.2 

408 

714 

Madison Strategies Group 

697 

0.142 

0.175 

7.0 

8.8 

350 

613 

Towards Employment 

698 

0.219 

0.167 

6.7 

8.4 

334 

585 

Flealth care 

351 

0.231 

0.233 

9.3 

11.7 

466 

816 

Manufacturing 

347 

0.278 

0.227 

9.1 

11.4 

454 

795 

Year 2 Survey sample 

Per Scholas 

552 

0.118 

0.199 

8.0 

10.0 

398 

697 

St. Nicks Alliance 

384 

0.101 

0.241 

9.6 

12.1 

482 

844 

Madison Strategies Group 

560 

0.113 

0.198 

7.9 

9.9 

396 

693 

Towards Employment 

562 

0.161 

0.192 

7.7 

9.6 

384 

672 

Flealth care 

295 

0.167 

0.265 

10.6 

13.3 

530 

928 

Manufacturing 

267 

0.201 

0.273 

10.9 

13.7 

546 

956 


SOURCE: MDRC calculations using PowerUP! tool. 

NOTES: MDE = minimum detectable effect; MDES = minimum detectable effect size; SD = standard deviation. 

a R-squared values differ by sample and site. For the administrative records sample, R-squared values are from 
the models for unemployment insurance earnings in Quarter 10. For the Year 2 Survey sample, R-squared values 
are from the models for current employment. 

b MDEs are for a two-tailed test at the 10 percent significance level with 80 percent power. 


The bottom half of Appendix Table A.9 shows the MDESs and MDEs for the Year 2 
Survey sample. Because the sample sizes here are smaller (as mentioned, around 80 percent of 
the full research sample responded to the Year 2 Survey at each site), the MDESs are larger than 
those for the administrative records sample. The MDESs here range from 0.192 (at Towards 
Employment) to 0.241 (at St. Nicks Alliance). For a continuous measure, such as earnings, with 
a standard deviation of $3,500, these MDESs translate into MDEs of $672 and $844, respec- 
tively. 


age points. The further the variance is from 0.5, the smaller the MDE. For example, if the control group level 
for a measure is 20 percent, the MDE for a study powered at 80 percent would be 8 percentage points. 
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While Towards Employment has the lowest MDES of the sites for both the administra- 
tive records and Year 2 Survey sample, if its sample were to be split into the two targeted 
sectors, the MDES would increase to well above the 0.2 threshold. For the Year 2 Survey 
sample, the MDES is 0.265 for the health care sector sample and 0.273 for the manufacturing 
sector sample. Those MDESs translate into MDEs of 13.3 and 13.7, respectively, for percentage 
measures, assuming a 50 percent control group level. It would be more difficult to detect any 
statistically significant impacts at this level. Because of this, the research team decided not to do 
the analysis at Towards Employment at the sector level but instead to present the results at the 
site level. 

Unit of Analysis 

Based on the results of the power analysis described in the previous section, a decision 
was made that WorkAdvance impacts would be presented primarily at the site level. This 
decision was also based on the substantial variation in various provider-level factors. In the 
subgroup analyses, however, the decision was to pool the sites since there is not enough 
statistical power to support such analyses at the site level. 

Follow-Up Period 

The job training literature makes it clear that extended follow-up is necessary to capture 
the long-term impacts of training programs. 23 In this report, the entire sample as a whole has 
2.25 years of follow-up from UI wage data. The WorkAdvance Year 2 Survey covers 18 
months of common follow-up for the full respondent sample. For these reasons, the impacts 
presented in this report should be viewed as “interim” in nature. Efforts are under way to collect 
longer-term follow-up data. The data acquisition contracts and informed consent fomis were set 
up to enable the collection of at least five years of common follow-up from UI wage data, funds 
permitting. 

Estimation Strategy 

Results presented in WorkAdvance were based on “intent-to-treat” impact estimates. 
That is, the impacts were calculated by comparing all individuals in the WorkAdvance group 
with all individuals in the control group, regardless of whether or how long individuals were 
engaged in WorkAdvance services. The impact estimates were regression adjusted using 
background characteristics of the sample, including prior earnings, age, race or ethnicity, 
education, household characteristics, and year of random assignment. (Covariate selection is 
described in more detail below.) 


23 See Card, Kluve, and Weber (2010). 
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For impacts on binary (0/1) and continuous outcomes, such as earnings during a period, 
MDRC used estimated regression models of the following form, using ordinary least squares: 24 

Yi = a+ fiP, + dXi + St 

where: 

Y l = outcome measure for sample member z; 

P t = indicator of program group membership for sample member z (1 = program 
group member; 0 = control group member); 

Xj = a set of background characteristics for sample member i; 

£( = random error for sample member z; 

P = the estimate of the impact of the program on the average value of the outcome; 

a = the intercept of the regression; and 

8 = the set of regression coefficients for the background characteristics. 

Regression adjustment can increase the statistical precision of the estimated effects, 
helping to distinguish nonnal variation in outcomes from the effects of WorkAdvance group 
members’ sole access to the WorkAdvance training and employment services. 

Confidence Intervals, P-Values, Standard Errors, and Effect Sizes of 

Program Impacts 

Appendix Table A. 10 displays more detailed statistical data on MDRC’s impact esti- 
mates of key employment and earnings outcomes by site. These details are included to provide 
more information on the uncertainty associated with specific impact estimates; they may be 
useful to meta-analysts who are interested in including the WorkAdvance findings. For each 
measure, the first two columns of each site panel show the lower and upper bounds of 90 
percent confidence intervals around the average impact estimate. The third column shows the 
level of statistical significance (p-value), using the results shown in the tables in Chapter 5, and 
the fourth column shows the standard error for each measure. The rightmost column in each site 


24 As sensitivity tests for several previous studies, MDRC estimated outcomes and impacts on binary (0/1) 
measures using ordinary least squares regression and again using logistic regression. MDRC found that the 
results were nearly identical. 
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The WorkAdvance Study 
Appendix Table A. 10 

Confidence Intervals, P-Values, Standard Errors, and Effect Sizes for Key Impacts, by Site 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 

90 Percent 90 Percent 90 Percent 90 Percent 

Confidence P- Effect Confidence P- Effect Confidence P- Effect Confidence P- Effect 


Outcome 

Interval 

Value 

SE 

Size 

Interval 

Value 

SE 

Size 

Interval 

Value 

SE 

Size 

Interval 

Value 

SE 

Size 

Among Year 2 Survey respondents 

All jobs (%) 

Ever employed 0.7 8.2 

0.051 

2.3 

0.17 

-5.8 

4.2 

0.791 

3.1 

0.03 

-1.5 

4.5 

0.412 

1.8 

0.07 

-2.8 

6.2 

0.530 

2.7 

0.05 

Currently employed 

3.7 

16.4 

0.010 

3.9 

0.22 

-8.2 

8.0 

0.984 

4.9 

0.00 

-0.4 

11.6 

0.122 

3.6 

0.13 

-0.2 

12.8 

0.109 

3.9 

0.13 

Percentage of months 
employed" 1 

-0.8 

8.4 

0.169 

2.8 

0.12 

-5.1 

7.0 

0.797 

3.7 

0.03 

-6.7 

2.7 

0.491 

2.9 

0.06 

-10.0 

-0.1 

0.096 

3.0 

0.13 

Employed 6 or more 
consecutive months" 1 

1.0 

13.9 

0.058 

3.9 

0.16 

-1.1 

15.2 

0.155 

5.0 

0.15 

-8.5 

3.3 

0.470 

3.6 

0.06 

-10.7 

2.2 

0.279 

3.9 

0.09 

Current or most recent job (%) 

Employed in targeted sector 34.4 

47.0 

0.000 

3.8 

0.83 

4.5 

19.3 

0.008 

4.5 

0.27 

9.7 

23.3 

0.000 

4.1 

0.33 

9.6 

23.2 

0.000 

4.1 

0.33 

Earnings 

Average total earnings 
per week ($) 

48 

148 

0.001 

30 

0.28 

-46 

77 

0.679 

37 

0.04 

-16 

69 

0.303 

26 

0.09 

-18 

56 

0.393 

23 

0.07 

Average hourly wage ($) 

1.21 

3.55 

0.001 

0.71 

0.28 

-1.67 

1.12 

0.749 

0.85 

0.03 

-0.36 

1.27 

0.358 

0.50 

0.08 

-0.49 

1.16 

0.505 

0.50 

0.06 

Employed and hourly wage 
above $12.00 (%) 

9.3 

23.3 

0.000 

4.3 

0.33 

-6.2 

11.1 

0.645 

5.3 

0.05 

4.2 

18.3 

0.009 

4.3 

0.22 

-4.4 

7.9 

0.636 

3.7 

0.04 

Employed and hourly wage 
above $15.00 (%) 

9.5 

22.8 

0.000 

4.1 

0.33 

-6.8 

8.4 

0.860 

4.6 

0.02 

0.9 

12.2 

0.059 

3.5 

0.16 

-7.1 

1.4 

0.274 

2.6 

0.09 

Hours 

Average hours worked 
per week (#) 

-1.0 

3.2 

0.399 

1.3 

0.07 

-0.9 

4.7 

0.276 

1.7 

0.11 

-0.7 

3.6 

0.268 

1.3 

0.09 

-0.9 

3.9 

0.304 

1.4 

0.09 

Employed full-time b (%) 

-1.2 

12.3 

0.178 

4.1 

0.12 

-4.2 

11.9 

0.436 

4.9 

0.08 

-2.6 

8.4 

0.387 

3.4 

0.07 

-0.1 

13.5 

0.103 

4.1 

0.14 

Sample size (total = 2,058) 

552 





384 





560 





562 






(continued) 
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Appendix Table A.10 (continued) 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 

90 Percent 90 Percent 90 Percent 90 Percent 

Confidence P- Effect Confidence P- Effect Confidence P- Effect Confidence P- Effect 
Outcome Interval Value SE Size Interval Value SE Size Interval Value SE Size Interval Value SE Size 

Among full research sample 
Quarter 2 to 10 


Ever employed (%) 

-1.2 

6.9 

0.247 

2.5 

0.09 

-1.1 

9.3 

0.198 

3.2 

0.12 

-3.2 

3.1 

0.993 

1.9 

0.00 

2.2 

9.5 

0.009 

2.2 

0.19 

Average quarterly 
employment 

0.1 

8.2 

0.096 

2.5 

0.12 

-3.5 

6.5 

0.619 

3.0 

0.04 

-1.6 

6.3 

0.319 

2.4 

0.07 

-0.8 

6.9 

0.189 

2.3 

0.09 

Earnings ($) 

1,830 

8,542 

0.011 

2,040 

0.18 

-4,349 

3,948 

0.937 

2,522 

0.01 

-612 

5,717 

0.185 

1,924 

0.09 

-599 

4,217 

0.217 

1,464 

0.08 

Year 1 (Quarters 2 to 5) 

Ever employed (%) 

0.8 

11.4 

0.058 

3.2 

0.14 

1.4 

14.0 

0.044 

3.8 

0.18 

-1.2 

6.8 

0.255 

2.4 

0.09 

0.2 

9.5 

0.089 

2.8 

0.12 

Average quarterly 
employment 

-2.5 

6.8 

0.443 

2.8 

0.06 

-3.2 

7.6 

0.509 

3.3 

0.06 

-0.4 

8.1 

0.140 

2.6 

0.11 

-3.7 

4.7 

0.838 

2.6 

0.01 

Employed in all 4 quarters 

-8.8 

1.8 

0.281 

3.2 

0.08 

-13.1 

-0.6 

0.070 

3.8 

0.16 

-4.7 

7.3 

0.719 

3.6 

0.03 

-9.5 

0.9 

0.177 

3.2 

0.09 

Earnings ($) 

-1,119 

1,420 

0.846 

772 

0.01 

-1,908 

1,402 

0.802 

1,006 

0.02 

-990 

1,647 

0.682 

802 

0.03 

-997 

1,022 

0.984 

614 

0.00 

Earned $20,000 or more 

-4.5 

3.8 

0.896 

2.5 

0.01 

-9.2 

1.3 

0.215 

3.2 

0.11 

-2.1 

7.7 

0.342 

3.0 

0.07 

-6.8 

1.1 

0.241 

2.4 

0.08 

Year 2 (Quarters 6 to 9) 

Ever employed (%) 

1.0 

11.0 

0.050 

3.1 

0.15 

-4.5 

8.6 

0.600 

4.0 

0.05 

-7.2 

2.2 

0.384 

2.9 

0.06 

0.4 

10.4 

0.076 

3.0 

0.13 

Average quarterly 
employment 

0.8 

10.6 

0.058 

3.0 

0.14 

-4.2 

8.1 

0.596 

3.7 

0.05 

-4.0 

5.6 

0.789 

2.9 

0.02 

0.4 

9.8 

0.075 

2.9 

0.12 

Employed in all 4 quarters 

0.9 

13.1 

0.062 

3.7 

0.14 

-1.7 

12.6 

0.207 

4.3 

0.11 

-3.2 

8.8 

0.443 

3.7 

0.06 

-0.8 

10.6 

0.161 

3.5 

0.10 

Earnings ($) 

1,780 

5,714 

0.002 

1,196 

0.22 

-2,128 

2,510 

0.892 

1,410 

0.01 

101 

3,574 

0.082 

1,056 

0.12 

238 

3,004 

0.054 

841 

0.13 

Earned $20,000 or more 

2.9 

14.2 

0.014 

3.5 

0.18 

-1.7 

11.3 

0.224 

3.9 

0.10 

3.7 

15.1 

0.007 

3.5 

0.19 

0.6 

10.5 

0.066 

3.0 

0.13 
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Appendix Table A.10 (continued) 


Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 


90 Percent 


90 Percent 


90 Percent 


90 Percent 



Confidence 

P- 

Effect Confidence 

P- 

Effect Confidence 

P- 

Effect Confidence 

P- 

Effect 


Outcome 

Interval Value 

SE 

Size 

Interval 

Value 

SE 

Size 

Interval 

Value 

SE 

Size 

Interval 

Value 

SE 

Size 

First quarter of Year 3 

Ever employed (%) 

0.3 11.8 0.084 

3.5 

0.13 

-10.2 

4.3 

0.508 

4.4 

0.06 

-2.8 

9.1 

0.383 

3.6 

0.07 

-0.8 

10.5 

0.156 

3.4 

0.10 

Earnings ($) 

693 1,885 0.000 

362 

0.25 

-851 

573 

0.748 

433 

0.03 

-110 

883 

0.200 

302 

0.09 

-215 

567 

0.460 

238 

0.05 

Sample size (total = 2,564) 

690 



479 





697 





698 






SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New York 
State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: SE = standard error. 

This table shows the upper and lower bounds of 90 percent confidence intervals around the average impact and the p-values, standard errors, and effect sizes 
of the impact estimates. Effect sizes are shown as absolute values and were calculated by dividing the impact estimate by the standard deviation of the pooled 
sample of sample members from both research groups. 

Sample sizes may vary because of missing values. 

a Measures reported in time intervals cover only the first 1 8 months following each sample member's month of random assignment (the common follow-up 
period). 

b Full-time employment is defined as working 35 hours or more per week. 



panel displays the effect sizes in absolute values. For each measure, the effect size was calculat- 
ed by dividing the impact estimate by the standard deviation for the pooled sample. As dis- 
cussed above, effect sizes standardize impact estimates for comparison with impact estimates 
from other studies. 

For example, the table shows that the 90 percent confidence interval around the impact 
on hourly wages in Per Scholas ranges from a lower bound of $ 1 .2 1 to an upper bound of $3.55 
per hour. Because both of these bounds are on the same side of zero (in other words, even the 
lower bound shows a positive increase), the estimate is statistically significant. This is also 
shown by the p-value, which indicates the probability of measuring an impact of this size or 
larger if there really is no true impact at the Per Scholas site. Because this probability is very 
low (p-value = 0.001), it can be concluded that the estimated effect is due to the true impact of 
Per Scholas rather than to chance. The standard error reflects the statistical uncertainty associat- 
ed with this estimate, factoring in the sample size, the standard deviation, and the units of 
measurement. The final element, the effect size (0.28), indicates that this is a moderate-sized 
impact based on statistical literature on effect sizes. 25 

Assessment of Possible Effects of Multiple Comparisons 

In recent years, the issue of multiple test bias has become more prominent in both the 
academic literature and the field of program evaluation more generally. The basic issue is well 
known and not new. Every time one estimates an impact on an outcome there is a precisely 
defined probability (conventionally 10 percent in such studies as WorkAdvance) of concluding 
that a program has had a true impact when the observed difference is simply due to chance. 
Since researchers typically examine many outcomes, the probability that at least one estimate 
will be statistically significant simply by chance can get very high. 

In the past, this issue was dealt with through researcher discretion. Impact analysts 
would assess patterns of findings and were trained not to emphasize small, sporadic effects even 
if they were statistically significant. As the technology used to produce outcome estimates has 
improved, however, it has become possible to produce thousands of estimates for a given report 
and to “cut the data” in innumerable ways. In addition, there is a thriving literature about how 
some studies have abused the statistical inference process by either “cherry picking” significant 
results or overemphasizing results that are probably due to chance. (This is sometimes called 
“p-value hacking.”) So there is broad consensus that something needs to be done to discipline 
the process. There is substantial debate, however, about what should be done. 


25 See Hill, Bloom, Black, and Lipsey (2007). As many authors have noted, effect sizes are domain de- 
pendent. This is a rather large effect size compared with that of other employment programs. 
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Confirmatory Measures 

One approach to dealing with this problem is to conduct fewer impact estimates and to 
state in advance which tests will be conducted. This was one strategy used in WorkAdvance to 
address the multiplicity problem. Evaluations such as WorkAdvance collect rich data that can 
be useful to diverse audiences. A common means of navigating this issue is to choose a select 
group of “confirmatory” outcomes, which are featured in the report and are highly influential in 
driving the determination of program effectiveness. Other measures (sometimes called “ex- 
ploratory” measures) are shown in the main report but do not have as much weight in defining 
“success.” Thus, the study can preserve the richness of the data while addressing the multiple 
comparisons problem. However, this approach requires that one think carefully about confirma- 
tory measures well in advance of measuring impacts. 

One consideration in choosing the confinnatory measures was a desire to capture the 
key goals of WorkAdvance programs: employment and earnings. Recent evidence has high- 
lighted the importance of measuring employment and earnings with multiple data sources. 26 
Another consideration is that with four sites, multiple comparisons adjustments can quickly get 
very conservative. Thus only two confirmatory measures were chosen in WorkAdvance. This 
results in eight statistical tests. In order to correct the p-values for an inflated familywise error 
rate, 27 the Westfall-Young correction was applied to the p-values. This correction was favored 
because of its perfonnance with correlated outcomes. 

Taking into account all these considerations, in WorkAdvance, one confirmatory out- 
come was chosen from the survey and one from administrative records. The measures chosen 
were participants’ current employment at the time of the Year 2 Survey interview and UI 
earnings in the first quarter of Year 3. 28 The Westfall-Young adjusted p-values are shown in 
Table 5.3 and are discussed in the main report. All other measures presented in the main report 
are exploratory measures. If there were no statistically significant estimated impacts on the 
primary measures, but there were impacts on some of the exploratory measures, this would 
imply that there is some preliminary evidence of the effectiveness of Work Advance. 


26 See Bamow and Greenberg (2015). 

~ 7 The familywise error rate is the probability of obtaining at least one statistically significant test across 
several measures due to chance (rather than the true impact of the program). 

^Initially the earnings measure chosen was earnings in Quarter 7. However, as more follow-up data be- 
came available, it was decided to stick with the concept of using the latest quarter of UI wage data as the 
confirmatory measure. 
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Categorical Measures 

Categorical measures can also exhibit the multiple comparisons problem when two ap- 
parent impacts represent the same actual effect. Because categorical measures are mutually 
exclusive, it follows that if more people are in one category, then fewer people must be in 
another category. For example, using the categorical version of hourly wages, if WorkAdvance 
increased the number of people with wages at the level of more than $15 per hour, then it also 
reduced the number of people at another wage level (for example, hourly wages of $8 or less). 
Impacts may be present in both of these levels simply due to the nature of the measures. 

To deal with this issue, two types of tests were run. First, the Westfall-Young correction 
was applied to the p-values for each level of categorical measures. Second, seemingly unrelated 
regression (SUR) models were run to test for differences in the distribution of categorical 
measures across research groups, and the F-test is used to test whether all program impacts are 
zero across the categories. For almost all categorical measures, the impacts were consistent 
across the two types of tests. 

Outliers 

To improve precision, when estimating program effects on the key continuous measures 
in the report (those that would be most affected by outliers, such as dollar-value measures), 
extreme values were identified as outliers, and for some measures, outliers were recoded to the 
value at the 99th percentile (for example, in the case of hourly wages and weekly earnings from 
the survey) or set to zero (for example, in the case of UI earnings and benefits). Appendix Table 
A.ll compares the impacts on income, wages, debt, and earnings measures by the level of 
exclusion of outlier values. The first set of columns shows the impacts with all values included. 
The second set of columns shows the impacts with extreme outliers removed (these are the 
impacts presented in the main body of the report). Not all measures shown in the table had 
extreme outliers, and for those measures, the impacts are the same in the first two sets of 
columns. (The details on changes made to outlier values are included in the footnotes of the 
table.) The third set of columns shows the impacts after excluding the top 1 percent of values. 

The table shows that the findings in this report are robust to outliers. If anything, the 
impacts are slightly larger after outliers are excluded. For example, the impact on Year 1 
earnings is — $39 for all responses, $11 once extreme outliers are removed, 29 and $299 after 
excluding the top 1 percent of values. 


29 UI earnings above $50,000 per quarter were set to $0. 
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The WorkAdvance Study 
Appendix Table A.ll 


Comparison of Impacts on Income, Wages, Hours, Debt, and Earnings, by Level of Exclusion of Outlier Values 





All Responses 



Extreme Outliers Removed 11 


Top 1 Percent Excluded 

Outcome 

N 

SE 

WA 

C 

Difference 

(Impact) 

N 

SE 

WA 

C 

Difference 

(Impact) 

N 

SE 

WA 

C 

Difference 

(Impact) 

Year 2 Survey 

Respondent income in 
prior month ($) 

1,889 

58 

1,435 

1,315 

l\g ** 

1,889 

58 

1,435 

1,315 

\\9 ** 

1,861 

46 

1,390 

1,168 

222 *** 

Household income in 
prior month ($) 

1,883 

72 

1,909 

1,860 

49 

1,883 

72 

1,909 

1,860 

49 

1,859 

61 

1,846 

1,724 

122 ** 

Weekly earnings ’ 5 ($) 

1,957 

15 

506 

461 

45 *** 

1,957 

14 

502 

459 

43 *** 

1,937 

13 

490 

442 

4 $ *** 

Hourly wage c ($) 

1,960 

0.36 

12.82 

11.96 

0.86 ** 

1,960 

0.31 

12.68 

11.87 

0.81 *** 

1,941 

0.27 

12.38 

11.45 

0 94 *** 

Hours worked per week (#) 

2,040 

0.7 

36.3 

35.0 

\ 4 ** 

2,040 

0.7 

36.3 

35.0 

\ 4 ** 

2,020 

0.7 

36.0 

34.4 

1.5 ** 

Amount spent/borrowed for 
primary training ($) 

2,051 

100 

198 

624 

-426 *** 

2,051 

100 

198 

624 

-426 *** 

2,029 

51 

119 

296 

-178 *** 

UnemDlovment insurance ($) 

Year 1 earnings d 

2,564 

395 

10,293 

10,332 

-39 

2,564 

393 

10,295 

10,284 

11 

2,539 

361 

10,009 

9,710 

299 

Y ear 2 earnings d 

2,564 

559 

15,715 

13,816 

1,899 *** 

2,564 

553 

15,713 

13,744 

1,969 *** 

2,539 

516 

15,296 13,004 

2,292 *** 

First quarter of Year 3 earnings d 

2,564 

163 

4,143 

3,667 

47 g *** 

2,564 

163 

4,143 

3,667 

476 *** 

2,539 

153 

3,999 

3,480 

519 *** 

Quarter 2-10 UI benefits^ 

1,866 

155 

1,754 

1,695 

59 

1,866 

155 

1,746 

1,681 

65 

1,848 

133 

1,588 

1,455 

134 
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Appendix Table A. 11 (continued) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from 
unemployment insurance (UI) administrative records from New York State Department of Labor, Ohio 
Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: N = number (sample size); SE = standard error; WA = WorkAdvance (program) group; C = 
control group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

a The impacts presented in the tables in the main body of the report are the impacts with extreme 
outliers removed. No extreme outliers were identified or removed from measures except as indicated. 

b Extreme weekly earnings outliers are defined as earnings over $2,000 per week. These outlier values 
were recoded to $2,000. 

c Extreme hourly wage outliers are defined as wages over $50 per hour. These outlier values were 
recoded to $50. 

^Extreme UI earnings outliers are defined as earnings over $50,000 per quarter. These outlier values 
were recoded to $0. 

e Extreme UI benefits outliers are defined as benefits over $6,500 per quarter. These outlier values 
were recoded to $0. 


Covariates 

Covariate selection for WorkAdvance was based on both theory and modeling. During 
covariate modeling, a series of regressions were run with earnings in Quarter 2 (from UI wage 
data) as the dependent variable. The LASSO (least absolute shrinkage and selection operator) 
method was used in the regressions to help protect against including spurious variables. 30 The 
variables tested were drawn from baseline and UI wage and benefits data. In addition to testing 
standard variables from these sources, several transformations were tested, including interaction 
terms, indexes, logs, and propensity scores. Variables with statistically significant coefficients 
and other variables correlated with the outcome were included in the final covariate models. 

The covariates were tested using the pooled WorkAdvance sample, and this sample was 
used to make decisions about the final covariate model. In the analysis, the impacts are present- 
ed by site, and the covariate model was adjusted somewhat for each individual site. The site 
covariate models were adjusted to include additional covariates where data were available for 
only some sites (for example, UI benefits data) and to eliminate unneeded covariates (such as 
the site dummies). Following is a list of the baseline characteristics that MDRC used as covari- 
ates in the regression models: 

• Random assigmnent year 

• WorkAdvance site 

30 The LASSO method is an alternative estimation method for linear regression models. The method min- 
imizes the residual sum of squares relative to a constant that is the sum of the absolute value of the coefficients. 
See Tibshirani (1996) for more information on the LASSO method. 
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• Sector 31 

• Gender 

• Race/ethnicity 

• Age 

• Educational attainment (varied by site) 

• Number of children living with respondent 

• Bom in state 

• Previous license or certificate in targeted industry 

• Employment status at baseline 

• Monthly family income 

• Employment and earnings history before random assignment 

• UI benefits history before random assignment 32 

To test the sensitivity of the regression adjustment, MDRC compared the adjusted and 
unadjusted research group means and differences (impacts) for key outcome measures (see 
Appendix Table A. 12). As shown, the adjusted and unadjusted estimates are very similar. These 
findings help confirm that the random assigmnent process resulted in the creation of research 
groups with similar characteristics and that the effort to field the WorkAdvance Year 2 Survey 
did not bias the results. 

Missing Data 

Covariates. Fifteen of the 21 covariates in the cross-site regression model for estimat- 
ing program impacts had nonmissing values for all study participants. The other six measures 
were collected from responses to the WorkAdvance baseline information form. These measures 
concern study participants’ gender, race/ethnicity, number of dependent children, previous 
receipt of credentials in the targeted sector, and family income. Among the full research sample, 
the proportion of missing responses for these measures ranged from about 0. 1 percent (for 
family income) to about 1.5 percent (for race/ethnicity). In response, missing values for covari- 
ates were imputed using the full sample’s mean, and dummy variables were added to the model 
to indicate missing status for each of the six covariates with missing values. 


31 A dummy variable for being randomly assigned in the health care sector was used as a covariate in To- 
wards Employment’s site-specific model. A dummy variable for being randomly assigned in the environmental 
remediation sector was used in St. Nicks Alliance’s site-specific model. 

32 UI benefits data were not available for sample members at Towards Employment, so this variable was 
not used as a covariate in Towards Employment’s site -specific model. 
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The WorkAdvance Study 
Appendix Table A. 12 


Adjusted Versus Unadjusted Impacts on Employment and Earnings, by Site 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Among respondents to Year 2 Survey 

Ever employed (%) 

4.4 * 

5.0 ** 

- 0.8 

-2.4 

1.5 

1.3 

1.7 

1.4 

Currently employed 

10.0 *** 

9 0 ** 

- 0.1 

- 2.1 

5.6 

4.6 

6.3 

5.3 

Percentage of months employed 2 

3.8 

2.6 

0.9 

- 1.1 

- 2.0 

-2.5 

-5.0 * 

- 6.2 * 

Employed 6 or more consecutive months ' 1 

7.5 * 

6.8 * 

7.1 

5.2 

- 2.6 

-2.7 

-4.3 

-4.8 

Current or most recent job 

Employed in targeted sector (%) 

40 7 *** 

42.0 *** 

229 *** 

23 2 *** 

16.5 *** 

14.8 *** 

16.4 *** 

17 7 * * * 

Average total earnings per week ($) 

9g *** 

j 2 g *** 

16 

18 

27 

18 

19 

22 

Average hourly wage ($) 

2 38 *** 

2.76 *** 

-0.27 

- 0.22 

0.46 

0.34 

0.33 

0.44 

Employed and hourly wage 
above $ 12.00 (%) 

Employed and hourly wage 

16.3 *** 

27 *** 

2.4 

1.9 

2 2 2 *** 

g 9 ** 

1.8 

2.6 

above $15.00 (%) 

16.2 *** 

16.0 *** 

0.8 

1.6 

6.6 * 

5.5 

-2.8 

-1.6 

Average hours worked per week (#) 

1.1 

1.8 

1.9 

1.2 

1.4 

1.1 

1.5 

1.2 

Employed full-time 13 (%) 

5.5 

7.2 * 

3.8 

2.8 

2.9 

2.3 

6.7 

6.1 

Sample size (total = 2,058) 

552 


384 


560 


562 


Among full research sample 

Quarters 2 to 10 

Ever employed (%) 

2.9 

3.8 

4.1 

4.1 

0.0 

-0.4 

3 9 *** 

6.6 *** 

Average quarterly employment (%) 

4.2 * 

3.9 

1.5 

1.2 

2.4 

1.5 

3.1 

3.7 

Earnings ($) 

5,186 ** 

5,741 *** 

-201 

-430 

2,552 

1,093 

1,809 

2,454 
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Appendix Table A. 12 (continued) 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Year 1 (Quarters 2 to 5) 

Ever employed (%) 

6.1 * 

6.1 * 

7 y ** 

7.6 * 

2.8 

2.6 

4.8 * 

5.4 * 

Average quarterly employment 

2.2 

0.9 

2.2 

1.7 

3.8 

3.0 

0.5 

1.2 

Employed in all 4 quarters 

-3.5 

- 6.1 * 

-6.9 * 

- 8.0 ** 

1.3 

0.2 

-4.3 

-3.7 

Earnings ($) 

150 

121 

-253 

-448 

328 

-306 

12 

307 

Earned $20,000 or more 

-0.3 

-0.3 

-4.0 

-4.9 

2.8 

0.8 

- 2.8 

-1.4 

Year 2 (Quarters 6 to 9) 

Ever employed (%) 

6.0 ** 

6.9 ** 

2.1 

2.2 

-2.5 

-3.3 

5.4 * 

6.0 * 

Average quarterly employment 

5.7 * 

6.0 ** 

2.0 

1.7 

0.8 

-0.3 

5.1 * 

5.8 * 

Employed in all 4 quarters 

7.0 * 

7.0 * 

5.5 

4.6 

2.8 

1.5 

4.9 

5.4 

Earnings ($) 

3,747 *** 

4,168 *** 

191 

172 

1,837 * 

1,151 

1,621 * 

1,920 ** 

Earned $20,000 or more 

8.5 ** 

g 7 *** 

4.8 

4.0 

q 4 *** 

7 1 ** 

5.5 * 

6.3 * 

First quarter of Year 3 

Ever employed (%) 

6.0 * 

74 ** 

-2.9 

-3.0 

3.1 

2.7 

4.9 

5.2 

Earnings ($) 

1,289 *** 

1,451 *** 

-139 

-154 

387 

248 

176 

226 

Sample size (total = 2,564) 

690 


479 


697 


698 



SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New York 
State Deparment of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: Sample sizes may vary because of missing values. 

a Measures reported in time intervals cover only the first 18 months following each sample member's month of random assignment (the common follow-up 
period). 

b Full-time employment is defined as working 35 or more hours in a week. 



Outcomes. Sample members with missing values for dependent variables (outcomes) 
were excluded from the impact estimates. Appendix Table A. 13 shows the percentage of 
sample members with missing values on key outcomes from both the Year 2 Survey and 
administrative records (there are no missing values for administrative records data). Procedures 
for assessing the effect of missing survey data are described above. 

Subgroup Analysis 

Impacts were calculated for key subgroups to better understand what works best for 
whom. In recent years, the standards for subgroup analysis have tightened. Leading methodolo- 
gists have argued for prespecification of subgroups and limiting the number of subgroups 
tested. 33 Following that lead, on the WorkAdvance project, MDRC selected two “confirmatory” 
subgroups: random assignment cohort and labor market attachment (level of prior employ- 
ment). Additional “exploratory” subgroups were examined as well but are not highlighted in the 
report. The following text is from the WorkAdvance analysis plan, which was completed well 
before impacts were estimated. 34 

Random Assignment Cohort 

MDRC chose random assignment cohort as one key subgroup, because start-up of the 
WorkAdvance program was slow across all the providers and the programs matured over time. 
Thus, the quantity and quality of services provided to participants who entered the program 
earlier compared with those who entered the program later likely differed. It was hypothesized 
that WorkAdvance would be most effective for the late cohort due to program maturation. The 
decision in choosing a cutoff point between the early and late cohorts was influenced by several 
factors: 

• The sites changed some eligibility criteria and recruitment strategies over 
time; this may have influenced the baseline characteristics of study partici- 
pants enrolling at a specific time. 

• In fall 2012, Towards Employment and the Madison Strategies Group both 
shifted their recruitment efforts and programs from a focus on the placement- 
first track to a focus on the training-first track. 

• St. Nicks Alliance added new training programs in hazard materials transpor- 
tation and pest control in mid-2012 due to labor market demand. 


33 Bloom and Michalopoulos (2010). 

34 The text was edited for clarity (for example, to spell out acronyms). 
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The WorkAdvance Study 
Appendix Table A.13 


Percentage Missing on Key Outcomes, by Site 



Per Scholas 

St. Nicks Alliance 

Madison 
Strategies Group 

Towards 

Employment 

Outcome (%) 

WA 

c 

WA 

c 

WA 

c 

WA 

c 

Among resnondents to Year 2 Survey 

Ever employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.4 

1.0 

0.0 

Currently employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.4 

1.0 

0.0 

Employed in targeted sector 

0.0 

0.4 

0.5 

1.1 

0.3 

0.8 

1.4 

1.1 

Average hourly wage at current or 

most recent job 

2.1 

9.1 

4.4 

7.3 

2.4 

4.9 

5.6 

3.6 

Currently or most recently working 

with hourly wages $15.01 or higher 

2.1 

9.1 

4.4 

7.3 

2.4 

4.9 

5.6 

3.6 

Employed full-time at current or most 

recent job 

0.3 

1.5 

0.5 

0.6 

0.7 

0.8 

1.7 

0.7 

Offered many opportunities for career 

advancement 

0.3 

1.1 

1.5 

0.6 

0.7 

1.1 

2.4 

0.4 

Sample size (total = 2,058) 

287 

265 

205 

179 

297 

263 

286 

276 

Among full research samnle 

Quarters 2 to 10 

Ever employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Average quarterly employment 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earnings 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Year 1 (Quarters 2 to 5) 

Ever employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Average quarterly employment 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Employed in all 4 quarters 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earnings 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earned $20,000 or more 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Year 2 (Quarters 6 to 9) 

Ever employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Average quarterly employment 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Employed in all 4 quarters 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earnings 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earned $20,000 or more 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

First quarter of Year 3 

Ever employed 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Earnings 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Sample size (total = 2,564) 

349 

341 

242 

237 

353 

344 

349 

349 


SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 
insurance administrative records from New York State Department of Labor, Ohio Department of Jobs and Family 
Services, and Oklahoma Employment Security Commission. 

NOTE: WA = WorkAdvance (program) group; C = control group. 
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• Most of the sites did not begin implementing postemployment services until 
late 2012 or early 2013. 

• There was a desire for a close to 50/50 split between the two groups. 

Accounting for all the above factors, the research team decided on a cutoff point of Oc- 
tober 2012 for the cohort definition. The early cohort includes all participants randomly as- 
signed through September 2012, and the late cohort includes all participants randomly assigned 
in or after October 2012. 

Labor Market Attachment 

Various MDRC studies have used participant level of disadvantage as a subgroup con- 
struct, and variation in impacts is often measured in this domain. The Employment Retention 
and Advancement Study (ERA) focused specifically on labor market experience, and the ERA 
research team selected prior employment status as a confinnatory subgroup. 35 ERA found 
substantial variation in impacts by the level of participants’ prior employment experience. 
Impact analyses showed that the programs produced positive economic effects (albeit relatively 
modest) for participants who entered ERA with a medium level of labor force attachment 
(individuals who had worked in two or three quarters of the prior year or had earnings between 
$3,000 and $10,000 that year), while no impacts were found for participants with a low or high 
level of labor force attachment. 36 

This second key subgroup stems from the hypothesis that WorkAdvance programs will 
be more effective for those with at least semiattachment to the labor market, as was seen in 
ERA. The positive effects seen in this middle group may have been because individuals who 
had not worked recently (those in the low recent labor force attachment subgroup) had too many 
barriers to employment to benefit from program services, while those same services offered too 
little value to individuals with extensive recent employment (those in the high recent labor force 
attachment subgroup). Additionally, with the recent recession, there has been a concern about 
the effects on reentry into the labor market for the long-term unemployed. 

The prior employment subgroup analysis breaks the sample into the following groups: 

• Unattached or long-term unemployed — sample members who have never 
been employed or who were not employed for at least seven months before 
random assignment 

35 Hamilton and Scrivener (2012b). 

36 Sample members from all the ERA model tests were combined and, based on their employment and 
earnings in the previous year, were put into three subgroups: high, medium, or low recent labor force attach- 
ment. 
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• Semiattached — sample members not employed for one to six months before 
random assigmnent 

• Fully attached — sample members working at the time of random assign- 
ment or not employed for less than one month before random assigmnent 


Conditional Subgroup Analysis 

An additional technique used for exploratory analysis is called “conditional subgroup 
analysis.” In this type of analysis, interaction temis are used to control for the moderating 
effects of other baseline characteristics when estimating the relationship between a particular 
subgroup and program effects. For example, in estimating whether WorkAdvance has a larger 
effect for the early or late cohort sample, it might be important to control for work experience in 
the sector or prior education. 37 


The conditional subgroup model specifies earnings and advancement outcomes as fol- 
lows: 


+x 

k 

where participant characteristics are measured in 
mean and: 


Y t =cc+pp t + ) S k CC, 


I 


J ki 


y k CC ki Pi + e t 

terms of deviations from the grand 


Y t 

Pi 

OC 

P 

S k 

Yk 

£i 


outcome for sample member i 

indicator of program group membership for sample member i (1 = program 
group member; 0 = control group member) 

client characteristics k for sample member i 

mean outcome for typical control group member 

program impact for typical program group member 

effect of client characteristic k on control group mean outcome 

effect of client characteristic k on program impacts 

random error for sample member / 


37 Controlling for other factors, including economic conditions, may also be important here, but it was not 
possible in this analysis. 
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Comparison of Survey and Unemployment Insurance 

Employment Impacts 

There are several reasons why estimates from UI administrative records may differ 
from estimates from survey data. This section first describes the strengths and weaknesses of 
each data source and then describes the results of a comparison between findings from the UI 
wage administrative data and the survey data in WorkAdvance. 

Administrative records have a number of advantages: 

• They cover the full sample, which enhances statistical power. 

• They contain a history of events. Administrative data can be used to produce 
time trends of employment, earnings, and public assistance — often at a fine 
level of granularity. 

• Measures created from administrative records are not affected by recall error, 
such as would be expected if a survey participant is asked to remember dis- 
tant past events. 

• They can often be collected from one centralized source, rather than by track- 
ing thousands of individuals. Once data collection procedures are in place, 
further data collection has low marginal costs. 

The primary disadvantage of administrative records is that they typically do not cover 
all jobs. It is estimated that UI records cover approximately 90 percent of jobs, though rates 
might be lower for low- wage workers. 38 For example, administrative records will not have 
infonnation on jobs in the infonnal sector or jobs with the federal government. 33 Furthermore, 
state administrative records do not have infomiation on what happens outside the state. While 
the conventional wisdom suggests that undercoverage in administrative records should be 
equivalent between research groups in randomized controlled trials (RCTs), it is easy to imagine 
cases where undercoverage could interact with intervention strategies to produce bias. 40 Differ- 
ing coverage rates could produce (or shroud) estimated impacts. 41 


38 Komfeld and Bloom (1999); Hotz and Scholz (2001). 

39 Komfeld and Bloom (1999). 

40 Bamow and Greenberg (2015). 

4l Riccio et al. (2013). This type of scenario at least partly accounted for the difference between survey- 
and administrative records-based estimates of employment in the New York City Conditional Cash Transfer 
experiment. In that study, there was evidence that program group members were disproportionately likely to 
seek employment in the informal sector, where jobs that qualified for program-related incentives were more 
easily found (such as babysitting). Such jobs are not typically covered by UI records. Therefore, the survey- 

(continued) 
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This may be especially troublesome in sectoral employment programs, including 
WorkAdvance, which by definition focus on tightly defined segments of the labor market . 42 If a 
sectoral program targets a sector with a different coverage rate in UI data compared with the 
overall labor market, there is the potential for biased impact estimates. As one example, within 
the transportation sector (the sector targeted by Madison Strategies Group), large trucking 
companies are often headquartered in one state but have employees working in many states. If 
more WorkAdvance group members end up in jobs within the transportation sector compared 
with control group members, it is possible that these in-sector, out-of-state jobs may be missed 
by administrative records. Moreover, a targeted sector might be more or less likely to have 
individuals who are self-employed, whose employment and earnings histories would not be 
included in administrative records. 

Surveys are also an important data source for many RCTs because they provide infor- 
mation that administrative records and other data fail to capture. Without survey data, it would 
be difficult to quantify program “dosage,” or the extent to which a person actually engaged with 
a program . 43 Survey data also provide valuable insight on certain behaviors, beliefs, program 
experiences, participant or household characteristics, and other issues that may influence 
outcomes observed in administrative records. In addition, summarized earnings data from 
administrative records can be better understood with survey data, which provide infomiation 
about work schedules, rates of pay, and job changes. Finally, in many domains, administrative 
records are not available, and some evaluations have to depend almost completely on surveys. 
(For example, the research team was interested in the impacts of WorkAdvance on various 
measures of overall well-being, which are not captured in administrative records.) 

Unlike administrative records — where data are obtained for the full study sample — 
survey data are relatively expensive to collect and often are obtained for only a subset of the full 
sample, with the expectation that they will represent the full sample, which in turn is expected to 
represent a larger population. When a survey fails to be representative, it is considered biased. 
The nonresponse analysis, described above, found little evidence that the survey results in 
WorkAdvance were biased. 

To summarize, both UI wage and survey data have advantages and disadvantages, and 
it is not clear which source is superior. Different results can be obtained based on the coverage 
and biases inherent in each data source and how these interact with the program and population 


based estimates of impacts were positive, whereas estimated impacts from administrative records were not 
statistically significant. 

42 Maguire et al. (2010). 

43 While most programs collect program participation data in management information systems, these data 
are typically not available for the control group. 
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The WorkAdvance Study 
Appendix Table A. 14 


Employment in a Ul-Covered Job Compared with Survey-Reported Employment at the Time of the Year 2 

Survey Interview, Among Survey Respondents, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strateiges Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

WA 

C 

Difference 

WA 

C 

Difference 

WA 

C 

Difference 

Currently employed according to survey 

74.4 

64.4 

10.0 *** 

65.3 

65.4 

-0.1 

77.6 

72.0 

5.6 

64.5 

58.2 

6.3 

Employed in quarter of survey interview 
according to UI 

72.1 

61.9 

10.2 ** 

63.7 

59.5 

4.2 

72.0 

71.2 

0.8 

70.8 

62.1 

g y ** 

Employment status coverage 

Same employment status 













according to survey and UI a 

77.5 

77.2 

0.3 

76.0 

77.3 

-1.3 

77.3 

75.7 

1.6 

81.5 

74.7 

6.8 * 

Currently employed only according to UI 

10.1 

10.2 

-0.1 

11.2 

8.4 

2.8 

8.6 

11.7 

-3.1 

12.5 

14.7 

-2.2 

Currently employed only according to survey 

12.4 

12.6 

-0.2 

12.8 

14.3 

-1.5 

14.1 

12.6 

1.5 

6.0 

10.6 

-4.6 * 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Survey employment is measured by month relative to each sample member's date of random assignment, while UI employment is measured in relative 
quarters. The UI employment measures are for the relative quarter in which the respondent was interviewed. This ranges from Quarter 6 to Quarter 1 1 . 

Four survey respondents were missing a current employment status and are not included in the first measure in the table. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a This measure includes both (1) sample members unemployed according to both the survey and UI and (2) sample members employed according to both the 
survey and UI. 



under study. Therefore, it is important to compare UI wage and survey data. Appendix Table 
A. 14 compares the employment rates of survey respondents in Ul-covered jobs with the survey- 
reported employment rates at the time of the Year 2 Survey. 

This analysis, shown in Appendix Table A. 14, helps answer a couple of questions. 
First, are the employment levels and effects consistent across these data sources? Second, for 
each survey respondent, do these data sources capture consistent employment information (that 
is, if a respondent said he was working at the time of the survey interview, was this reflected in 
his UI wage records)? The first two rows of the table answer the first question: Here the results 
show that, for three of four sites, the levels and effects on employment at the time of the survey 
interview are fairly consistent across data sources (though the effect sizes — which are not 
statistically significant — vary for St. Nicks Alliance and Madison Strategies Group). For the 
fourth site, Towards Employment, only the effects on Ul-based employment are statistically 
significant. In addition, for this site, more employment (at the time of survey interview) is 
captured via UI wage records than from the survey. 

The remaining rows of results provide insight on the second question. Here again, for 
three sites, the data sources reflect consistent employment infonnation on about three-fourths of 
the respondent samples. In the case of Towards Employment, it appears that WorkAdvance 
may have led to an increase in employment that is covered by UI wage records in Ohio. Overall, 
the results of this analysis suggest that, for the most part, the UI wage and survey data sources 
tell similar stories at each of the four sites. 
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The WorkAdvance Study 
Appendix Table B.l 

Additional Characteristics of Research Sample Members at Baseline 


Characteristic 

PS 

SNA 

MSG 

TE 

Overall 

Demographic characteristics (%) 

Citizenship 

Born in United States 

71.9 

76.9 

95.4 

97.9 

86.3 

Naturalized 

15.7 

11.9 

1.1 

2.0 

7.3 

Noncitizen 

12.5 

11.1 

3.4 

0.1 

6.4 

Veteran 

2.2 

6.4 

11.7 

4.6 

6.2 

Family status 

Living with a partner 3 (%) 

8.1 

8.4 

17.0 

8.3 

10.6 

Primary caregiver 5 (%) 

17.7 

24.8 

38.4 

43.3 

31.6 

Single caregiver 

7.4 

11.2 

13.4 

31.1 

16.2 

Noncustodial parent 1 

9.1 

22.9 

17.1 

10.3 

14.2 

Average number of children living with study participant 11 

0.3 

0.4 

0.8 

0.9 

0.6 

Age of primary caregiver's youngest child (%) 

Less than 6 years old 

9.4 

13.1 

24.4 

24.5 

18.3 

6 to 9 years old 

3.8 

4.2 

5.5 

7.6 

5.4 

10 to 15 years old 

2.8 

4.6 

7.3 

7.5 

5.6 

16 years or older 

1.7 

2.7 

1.1 

3.7 

2.3 

Not a primary caregiver for any children 

82.3 

75.4 

61.6 

56.7 

68.4 

Currently enrolled in education or training program (%) 

4.3 

3.0 

4.6 

7.6 

5.1 

English as a second language 

0.0 

0.2 

0.3 

0.0 

0.1 

Adult Basic Education 

0.0 

0.0 

0.1 

0.3 

0.1 

High school/GED preparation course 

0.0 

0.6 

1.0 

1.7 

0.9 

Occupational skills training 

0.7 

0.8 

0.9 

1.1 

0.9 

College course toward associate's/two-year degree 

2.3 

0.8 

1.1 

3.7 

2.1 

College course toward bachelor's/four-year degree 

1.9 

0.4 

1.1 

0.6 

1.1 

Other 

0.1 

0.2 

0.3 

0.3 

0.2 

Employment status (%) 

Average hourly wage at current or most recent job e 

Never employed 

3.6 

2.1 

0.7 

2.7 

2.3 

$7.24 or less 

5.9 

5.0 

5.5 

6.7 

5.9 

$7.25-$8.00 

11.3 

14.0 

13.8 

21.2 

15.2 

$8.01-$9.00 

12.3 

9.0 

14.5 

20.9 

14.6 

$9.01410.00 

13.2 

17.2 

20.9 

16.5 

16.9 

$10.01412.50 

17.1 

12.6 

25.7 

16.9 

18.5 

$12.51 -$15.00 

18.4 

15.5 

13.9 

8.0 

13.8 

$15.01 or higher 

18.1 

24.7 

5.0 

7.0 

12.8 

Among those unemployed for more than 12 months f 

40.0 

44.9 

25.7 

38.8 

40.1 
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Characteristic 

PS 

SNA 

MSG 

TE 

Overall 

Average hours worked per week at current 

or most recent job 

32.0 

34.4 

37.7 

33.1 

34.3 

Among those currently working 

22.6 

25.8 

30.6 

28.6 

28.0 

Circumstances that may affect job change or retention (%) 

Physical or mental health problem that limits work 

0.6 

0.8 

1.9 

1.0 

1.1 

Previously convicted of a crime and incarcerated 

4.8 

15.9 

26.8 

11.9 

14.8 

Other income sources (%) 

Earnings from spouse or partner 

9.9 

12.6 

22.3 

8.8 

13.5 

Medical coverage, among those with coverage (%) 

Type of health insurance plan 

Employer-provided 

1.9 

0.4 

16.5 

9.3 

6.2 

Other 

24.9 

7.1 

51.0 

18.2 

23.5 

Publicly funded coverage 

73.2 

92.5 

32.5 

72.5 

70.4 

Parents with publicly funded coverage for their children g 

53.5 

38.5 

68.4 

74.2 

62.3 

Housing arrangements (%) 

Public housing 

13.8 

9.9 

4.9 

11.7 

10.1 

Section 8 housing vouchers 

6.1 

3.6 

3.2 

5.0 

4.5 

Reduced rent 

2.9 

3.8 

1.3 

3.3 

2.7 

Group shelter 

1.5 

5.5 

1.9 

1.7 

2.4 

Homeless 

0.0 

0.2 

0.6 

1.1 

0.5 

Sample size 

690 

479 

697 

698 

2,564 


SOURCE: MDRC calculations from the WorkAdvance baseline information form. 

NOTES: PS = Per Scholas; SNA = St. Nicks Alliance; MSG = Madison Strategies Group; TE = 
Towards Employment; GED = General Educational Development certificate. 

Italic type indicates that the metric is not among the hill sample shown in the table. 

Sample sizes may vary because of missing values. 

a This does not include sample members who responded as "married and living with spouse." 
b A primary caregiver is a parent who has at least one child living with him or her more than half the 
time. 

c While there is a legal definition for a noncustodial parent, in this analysis a noncustodial parent is 
defined as a parent who has at least one child not living with him or her for more than half the time. 

d This is the average number of children living with the sample member for more than half the time. 
The estimate is based on how many people are in the family minus the sample member and another 
adult if he or she is married and living with the spouse or is not married but living with a partner. 
e Wages for sample members who have never been employed are counted as $0. 

‘This is restricted to sample members unemployed for more than 12 months whose wage at their 
most recent job was higher than $15 per hour. 

g The percentage of parents with publicly funded health care coverage for their children is measured 
among sample members with children. 
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The WorkAdvance Study 
Figure B.l 

Annual Unemployment Rates 2011-2015, Nationally and by Metropolitan Area 



2011 2012 2013 2014 2015 

SOURCE: Local Area Unemployment Statistics data from the Bureau of Labor Statistics. 

NOTE: Unemployment rates are shown by metropolitan area and for the United States. The Cleveland 
metropolitan area includes Elyria and Mentor, Ohio. The New York metropolitan area includes northern New 
Jersey and Long Island, New York. 
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The WorkAdvance Study 
Figure B.2 

Sector Employment in Quarter 3, 2011, and Quarter 3, 2013, by Metropolitan Area 

Cleveland, Ohio 

400.000 
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100,000 
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400.000 
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Tulsa, Oklahoma 
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Transportation and manufacturing 
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Figure B.2 (continued) 


SOURCE: U.S. Census Bureau, Quarterly Workforce Indicators from the Longitudinal Employer-Household 
Dynamics extraction tool. 

NOTES: Sectors and industries are defined by the North American Industry Classification System. Manufacturing 
includes codes starting with 31-33; health care includes codes starting with 62; environmental remediation includes 
codes starting with 236, 2371, 2373, 2379, 238, 5617, and 562; information technology includes codes starting 
with 5112, 517, 518, 5415, and 8112; and transportation includes codes starting with 48-49. 

The employment numbers presented are the total number of industry jobs on the first day of the quarter. 
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The WorkAdvance Study 
Appendix Table C.l 


Percentage Distribution of Applicant Recruitment Sources During Select Time Periods 




St. Nicks 

Madison 

Towards 

Recruitment Source 

Per Scholas a 

Alliance 3 

Strategies Group 

Employment 

Among all applicants (%) 





Friend or family member 

32.9 

31.7 

5.8 

35.0 

Internet 

22.9 

17.2 

47.9 

13.6 

Flyer, poster, newsprint 

12.9 

13.8 

NA 

11.0 

Job/career fair 

8.6 

10.3 

1.4 

NA 

Another program or organization 

24.3 

21.4 

17.8 

14.6 

Radio/TV 

1.4 

0.7 

0.5 

5.7 

Walk-in 

NA 

4.1 

NA 

5.4 

Employer 

1.4 

1.4 

0.5 

NA 

Other/unknown b 

0.0 

2.1 

26.2 

14.6 

Applicants (total = 1,783) 

70 

145 

432 

1,136 

Among applicants randomly assigned (%) 





Friend or family member 

28.6 

28.0 

8.6 

33.3 

Internet 

14.3 

30.0 

65.7 

11.1 

Flyer, poster, newsprint 

0.0 

20.0 

0.0 

5.3 

Job/career fair 

21.4 

2.0 

0.0 

0.0 

Another program or organization 

21.4 

14.0 

20.0 

9.9 

Radio/TV 

7.1 

0.0 

0.0 

6.2 

Walk-in 

NA 

8.0 

0.0 

5.3 

Employer 

7.1 

0.0 

0.0 

0.0 

Other/unknown b 

0.0 

0.0 

5.7 

28.8 

Applicants (total = 342) 

14 

50 

35 

243 


SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas (PS) and St. Nicks 
Alliance (SNA), a report based on program tracking data provided by Madison Strategies Group (MSG), and 
program tracking data provided by Towards Employment (TE). 

NOTES: NA= not applicable. 

At PS, the recruitment funnel analysis covered 3 weeks in July 2012, and applicants are individuals who 
attended orientation. 

At SNA, two rounds of the recruitment funnel analysis were completed in mid-2012 covering 15 total weeks. 

The findings from both rounds were combined in this analysis. Applicants are individuals who expressed an interest 
in WorkAdvance. 

At MSG, the recruitment funnel analysis covered 6 weeks in mid-2012, and applicants are individuals who were 
scheduled for orientation. 

At TE, the recruitment funnel analysis covered 64 weeks from mid-2011 to late 2012. Applicants are individuals 
from the 1 1 most common recruitment sources who expressed an interest in WorkAdvance. 

See Appendix Table C.2 for details regarding the length and timing of the data coverage period and the 
definition of an applicant. 

“Applicants were allowed to select more than one recruitment source at PS and in the second round of the 
recruitment funnel at SNA, so percentages may sum to more than 100 percent. 

Applicants at TE in the category of "other/unknown" did not specify how they heard about WorkAdvance. At 
MSG, this category includes applicants referred by training programs and job placements and applicants who did 
not specify how they heard about WorkAdvance. 
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The WorkAdvance Study 
Appendix Table C.2 

Criteria for Study and Program Eligibility, by Provider, During Select Time Periods 


Study intake 

Eligibility 

requirements 

• 18 years or older 

• Legally allowed to work in the United States 

• Monthly family income less than 200 percent of the federal poverty level 

• Earning less than $15 per hour, if employed at intake 

Program intake 


Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Assessment 

score 

• TABE: 1 Oth-grade level 
(lowered to 9.5 grade level 
in July 2012) 

• TABE: 9th-grade level 
(later lowered to 8th- 
grade level) 

• Prove It! Math and Reading: 
8th-grade level 

• Mechanical aptitude test 

• Behavioral assessment 

• TABE Locator: 6th- to 

1 Oth-grade level, depend- 
ing on track 

OST-specilic 

criteria 


• Driver’s license (HCDL 
only) 

• Manual dexterity test 

• Driver’s license (CDL only) 

• Criminal background 
check (health care only) 

• Sector screening 
questionnaire 

Additional 

eligibility 

requirements 3 

• Not eligible for another 
contract (e.g., veterans) 

• High school diploma/GED 

• Not trained by St. Nicks 
Alliance in past 2 years 

• Drug screen 


• Drug screen 


(continued) 




Appendix Table C.2 (continued) 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Intake 

process 

Over 1-7 days or longer: 

• Potential applicants attend 
orientation. Staff deter- 
mines basic eligibility. 

• Eligible applicants take the 
TABE. Staff interviews 
those who pass. 

• Applicants are interviewed 
again. Those who remain 
fill out paperwork and 
enter random assignment. 

Over 1-2 days or longer: 

• Staff schedules eligible 
applicants for orientation. 

• At orientation, applicants 
take the TABE. Those 
who pass are interviewed 
and fill out paperwork. 

• Applicants complete drug 
screening. 

• Staff conducts case 
conferences and sched- 
ules eligible applicants 
for random assignment at 
a later date. 

Over 2-3 days or longer: 

• Applicants complete 
paperwork and tests. Those 
who pass receive a blank 
work history template. 

• Applicants return with draft 
work history. Staff inter- 
views applicants and con- 
ducts case conferences. 

• Eligible applicants enter 
random assignment. 

Over 2 days or longer: 

• Staff checks eligibility via 
phone and schedules 
candidates for orientation. 

• At orientation, applicants 
complete all assessments 
and are interviewed by the 
staff. 

• Applicants complete drug 
screening (48 hours). 

• Those who pass fill out 
paperwork and enter 
random assignment. 


NOTES: TABE = Tests of Adult Basic Education; OST = occupational skills training; HCDL = hazmat commercial driver’s license; CDL = 
commercial driver’s license; GED = General Educational Development certificate. 
a Some additional eligibility requirements changed over time across all the providers. 
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The WorkAdvance Study 
Appendix Table C.3 


Description of Program-Tracking Metrics, by Site 


n 

Metric 

Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Table 2.1 

Ever participated in 
any career 
readiness activity 8 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended at least 1 
day of career readiness training 

Participant completed initial 
assessment to discuss career 
goals and employment barriers 

Ever started skills 
training 11 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 

Ever completed 
skills training 1 ’ 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Ever obtained a 

credential in 
targeted sector 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Table 2.3 

Ever participated in 
any career 
readiness activity 8 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended at least 1 
day of career readiness training 

Participant completed initial 
assessment to discuss career 
goals and employment barriers 

Ever started 

classroom-based 

career readiness 
training 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended first day of 
skills training program, which 
includes career readiness 
training 

Participant attended at least 1 
day of career readiness training 

Participant attended first day of 
contextualized career readiness 
training workshop 

Ever completed 
classroom-based 

career readiness 
training 

Participant attended last day of 
skills training program, which 
includes career readiness 
training 

Participant attended last day of 
skills training program, which 
includes career readiness 
training 

Participant completed at least 4 
days of career readiness training 

Participant successfully 
completed contextualized career 
readiness training workshop 

Ever received help 
obtaining 

supportive services 

Participant received a referral to 
an outside organization 

(Not measured) 

Participant received a referral to 
an outside organization, a 
transportation voucher, or a 
training voucher 

Participant met with staff to 
discuss barrier removal 


(continued) 
























265 


Appendix Table C.3 (continued) 



Metric 

Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 


Ever scheduled to 
start skills training 

All participants 

All participants 

Participant was designated prior 
to RA as "on the training track" 

Participant was signed up for 
training 


Ever started skills 
training 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 

Participant attended first day of 
skills training program 


Ever completed 
skills training 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Participant is reported by 
training instructor to have 
completed all requirements of 
training program 

Table 2.5 b 

Ever dropped out of 
skills training 

Participant started skills training 
and is reported by training 
instructor to have withdrawn or 
been dismissed from training 

Participant started skills 
training, did not complete 
training within 18 months of 

RA, and the participant's 
training cycle is complete 

Participant started skills training 
and is reported by training 
instructor to have withdrawn or 
been dismissed from training 

Participant started skills training 
and is reported by training 
instructor to have withdrawn or 
been dismissed from training 


Enrolled in skills 
training at 18 
months after RA 

Participant started skills 
training, did not complete skills 
training, and did not drop out of 
skills training within the first 18 
months of enrollment 

Participant started skills 
training, did not complete skills 
training, and did not drop out of 
skills training within the first 18 
months of enrollment 

Participant started skills 
training, did not complete skills 
training, and did not drop out of 
skills training within the first 18 
months of enrollment 

Participant started skills 
training, did not complete skills 
training, and did not drop out of 
skills training within the first 18 
months of enrollment 


Ever obtained a 

credential in 
targeted sector 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 

Participant passed required 
exam(s) for at least 1 of the 
credentials offered through the 
provider 


(continued) 
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n 

Metric 

Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Table 2.5 (continued) 11 

Average number of 
weeks in training 

For participants who completed 
training, the average number of 
weeks between training start and 
end dates; for participants who 
dropped out of training, the 
average number of weeks 
between training start and 
halfway point of training cycle; 
for participants currently 
enrolled in training, average 
number of weeks between 
training start and 1 8 months 
after enrollment; for participants 
who never started training, 0 
weeks 

For participants who completed 
training, the average number of 
weeks between training start and 
end dates; for participants who 
dropped out of training, either 
the average number of weeks 
between training start and 
halfway point of training cycle 
or average number of weeks 
between training start and 
dropout date, where dropout 
date is recorded; for participants 
currently enrolled in training, 
average number of weeks 
between training start and 18 
months after enrollment; for 
participants who never started 
skills training, 0 weeks 

For participants who completed 
training, the average number of 
weeks between training start and 
end dates; for participants who 
dropped out of training, the 
average number of weeks 
between training start and 
dropout date; for participants 
currently enrolled in training, 
average number of weeks 
between training start and 18 
months after enrollment; for 
participants who never started 
training, 0 weeks 

For participants who completed 
training, the average number of 
weeks between training start and 
end dates; for participants who 
dropped out of training, the 
average number of weeks 
between training start and 
dropout date; for participants 
currently enrolled in training, 
average number of weeks 
between training start and 18 
months after enrollment; for 
participants who never started 
training, 0 weeks 


SOURCES: Program tracking systems managed by Per Scholas, St. Nicks Alliance, Madison Strategies Group, and Towards Employment. 
NOTES: RA = random assignment. 

All metrics include only activities that started and occurred after random assignment. 

The metrics included are among participants and cover the first 18 months of program activity. 

a The career readiness activity metric captures the first activity after random assignment. 

b All skills training-related measures include only skills training programs offered through the provider. 




The WorkAdvance Study 
Appendix Table C.4 

Percentage Distribution of Reasons for Drop-Off Among Applicants 
Who Attended Orientation 


Description (%) 

Per Scholas 

St. Nicks 

Alliance 

Madison 
Strategies Group 

Towards 

Employment 

Reason for drop-off after orientation 

Did not meet eligibility requirements 2 

15.7 

0.0 

0.0 

0.0 

Left after orientation 

12.9 

12.4 

14.5 

7.7 

Did not pass assessment test(s) 

35.7 

22.7 

18.8 

21.1 

Did not attend staff interview* 5 

7.1 

1.0 

19.8 

NA 

Did not pass internal case conference 

8.6 

10.3 

28.5 

12.3 

Did not attend random assignment appointment 

0.0 

2.1 

1.4 

NA 

Applicants who attended orientation (total = 787) 

70 

97 

207 

413 


SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas (PS) and St. Nicks 
Alliance (SNA); a report based on program tracking data provided by Madison Strategies Group (MSG); and 
program tracking data provided by Towards Employment (TE). 

NOTES: NA = not applicable. 

Refer to the notes to Appendix Table C.l and to Appendix Table C.2 for provider-specific details regarding the 
length and timing of the data coverage period and the definition of an applicant, as there is variation across 
providers. 

Screening processes varied over time at all providers. 

initial checks for applicant eligibility were performed before orientation at SNA, MSG, and TE, and eligibility 
was checked after orientation at PS. All providers verified applicants' eligibility after orientation. 

b "Did not attend staff interview" was not used as a drop-off point in the analysis at TE because its data did not 
include this level of detail. 
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The WorkAdvance Study 
Appendix Table D.l 


Year 2 Impacts on Educational Attainment, Internships, On-the-Job Training, and Employment Services, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Educational attainment 

Less than a high school diploma 

0.4 

0.7 

-0.4 

8.0 

8.2 

- 0.2 

7.7 

7.7 

0.0 

6.0 

5.4 

0.6 

High school diploma 

19.6 

16.2 

3.3 

25.6 

25.6 

- 0.1 

32.1 

30.0 

2.1 

30.4 

32.6 

-2.3 

GED 

9.0 

10.3 

-1.3 

15.5 

15.9 

-0.5 

12.5 

11.5 

1.0 

10.0 

11.4 

-1.4 

Some college credit but less than 

1 year 

11.9 

10.6 

1.3 

9.9 

12.2 

-2.3 

10.5 

11.4 

-0.9 

12.9 

15.6 

-2.7 

1 or more years of college, but 
no degree 

21.7 

25.6 

-3.9 

20.3 

18.8 

1.4 

23.4 

22.0 

1.4 

24.5 

21.0 

3.6 

Associate's or 2-year college degree 

14.7 

12.1 

2.5 

8.4 

10.0 

-1.7 

7.9 

12.8 

-4.9 

8.5 

9.4 

-0.9 

Bachelor's degree or above 

22.8 

24.4 

- 1.6 

12.4 

9.3 

3.1 

6.0 

4.7 

1.3 

7.7 

4.7 

3.1 

Participated in any E&T or obtained 
an employment service 3 

95.9 

76.2 

19 7 *** 

97.9 

76.1 

21.8 *** 

99.0 

63.9 

25 1 *** 

93.9 

68.5 

25 4 *** 

Obtained an employment service * 3 

93.3 

65.4 

27 9 *** 

96.5 

67.8 

28 7 *** 

98.0 

50.7 

47 ^ *** 

90.9 

58.7 

22 \ *** 

Ever participated in an internship 

31.0 

15.1 

15 9 *** 

11.8 

7.2 

4.7 

11.0 

4.0 

7 o *** 

23.6 

4.9 

7 *** 

Currently participating in an internship 

2.6 

3.3 

-0.7 

1.5 

2.2 

-0.7 

2.2 

0.2 

1 9 ** 

0.7 

1.1 

-0.5 

Ever paid as part of an internship 

12.9 

7.9 

4.9 * 

4.6 

4.2 

0.4 

8.9 

3.3 

5 7 *** 

1.6 

2.3 

-0.7 

Ever worked in an on-the-job 
training position 

Currently working in an on-the-job 

4.0 

7.8 

-3.8 * 

5.9 

3.3 

2.6 

6.1 

2.2 

2 9** 

6.8 

2.0 

4 8 *** 

training position 

0.4 

0.4 

0.0 

3.0 

0.5 

2.4 * 

0.7 

1.1 

-0.4 

0.4 

1.1 

- 0.8 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 
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Appendix Table D.l (continued) 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; GED = General Educational Development certificate; E&T = education and training. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall- Young adjusted p-values were used for 
categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. No statistically significant differences were found. 

Sample sizes may vary because of missing values. 

a Measure includes job training services such as job-specific skills training, career readiness help, job search assistance, postemployment help, training in 
basic reading and math skills, college courses, short-term classes, internships, on-the-job training, and GED classes. 
b Measure includes career readiness services, job search assistance, and postemployment help. 



Appendix E 


Net Cost Thought Experiment 




This appendix uses a thought experiment to demonstrate the implications of dividing aggregate 
costs in Year 2 of the WorkAdvance program by the number of Year 2 program entrants. It is 
based on assuming that WorkAdvance was in a steady state not only in Year 2, but also in the 
two prior years. The exercise has two important implications: 

1. If the assumption about the steady state were valid, dividing gross aggregate costs 
in Year 2 by the number of program entrants during Year 2 would provide an ap- 
propriate estimate of gross costs per participant for those entering during Year 2, 
including the program costs they incurred after Year 2. 

2. Because the assumption is invalid — Year 1 began in April 2011, but program en- 
try did not begin until well after that date and program start-up was slow — aggre- 
gate costs in Year 2 are understated relative to what they would be under a true 
steady state. As implied by the thought experiment, however, most of this under- 
statement would occur in the estimated cost of postemployment retention and ad- 
vancement because that activity occurs late in the program process. To adjust for 
this understatement, the estimated Year 2 cost of postemployment services is re- 
placed by the Year 3 cost of these services, as explained below. 

The thought experiment is as follows: 

If WorkAdvance were in the steady state not only in Year 2 but also in the two prior 
years, someone entering WorkAdvance in the first month of Year 2 could potentially incur costs 
during every month of Year 2 plus every month of Year 3, given the 24 months of eligibility. 
Year 2 costs represent only the first 12 months of the 24. However, someone else who entered 
WorkAdvance in the first month of Year 1 could potentially incur costs during all 12 months of 
Year 2, and these costs would be included as part of Year 2 costs. (Notice that because Work- 
Advance started after the first month of Year 1, this second person does not actually exist. This 
causes Year 2 costs to understate costs incurred during the last 12 months of the 24 months of 
eligibility. These missing costs should mainly be the cost of postemployment services.) 

Now, look at someone entering WorkAdvance in the second month of Year 2. That per- 
son could potentially incur costs for 1 1 months of Year 2 plus 13 months after Year 2. Costs in 
Year 2 do not capture these 13 months of costs. However, someone who entered WorkAdvance 
in the second month of Year 1 could potentially incur costs during all months of Year 2, and 
someone entering in the second month of Year 0 (the hypothetical year before Year 1) could 
potentially incur costs for the first month of Year 2. This would account for all 13 months of the 
costs that are missed for the person entering in Month 2 of Year 2. 

Obviously, this exercise could be continued for a person entering in the third month of 
Year 2, entering in the fourth month, and so forth. Consider just one more possibility: a person 
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who enters WorkAdvance in the twelfth month of Year 2. This person would incur costs for 
only one month during Year 2, but he could potentially incur costs for 23 months after Year 2. 
However, given the steady-state assumption, the potential Year 2 costs for 12 of these months 
would be accounted for by an individual entering in the twelfth month of Year 1, and the 
additional 1 1 months would be accounted for by another individual entering in the twelfth 
month of Year 0. (Notice that the second of these individuals does not exist, but the first may 
because WorkAdvance was well under way by the twelfth month of Year 1. Thus, most of the 
missing costs would be incurred late in the 24 months of eligibility and, consequently, would 
mainly consist of postemployment services.) 

Under the assumption about the steady state, therefore, total aggregate costs in Year 2 
would account for all the costs incurred by all those entering WorkAdvance in that year, 
including the costs they incurred after the end of Year 2. Thus, if the assumption were valid, 
dividing aggregate costs for Year 2 by entrants in Year 2 would provide an estimate of costs per 
participant for those entering WorkAdvance during Year 2 over their 24 months of eligibility. 

But in fact, Year 1 was nowhere near the steady state and Year 0 does not exist. Be- 
cause the assumption is invalid, total costs in Year 2 are understated somewhat, particularly 
postemployment costs relative to the steady state. As mentioned in Chapter 4 and Appendix A, 
to adjust for this, Year 3 postemployment costs are used instead of Year 2 postemployment 
costs in computing costs per participant in the steady state. Year 3 is not used to represent the 
steady state because there was a break in program recruitment after random assigmnent ended 
in that year, leading to the understatement of other costs relative to the steady state. 
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The WorkAdvance Study 
Appendix Table F.l 

Year 2 Impacts on Current Job Characteristics, by Site 

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 

Difference Difference Difference Difference 


Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) 


Employment status 

Currently employed (%) 

In targeted sector 

74.4 

48.2 

64.4 

16.2 

10.0 *** 

22 1 *** 

65.3 

21.7 

65.4 

12.7 

- 0.1 
g 1 ** 

77.6 

55.5 

72.0 

36.5 

5.6 

19 0 *** 

64.5 

37.8 

58.2 

23.1 

6.3 

14 7 *** 

Average number of months in 
current job 

5.6 

5.1 

0.5 

5.0 

4.6 

0.4 

6.2 

5.9 

0.3 

4.8 

5.3 

-0.5 

Earnings 

Average total earnings per week ($) 

460 

341 

j Yg *** 

389 

376 

13 

469 

404 

64 ** 

275 

249 

26 

Average hourly wage ($) 

12.26 

9.25 

3.01 *** 

9.53 

9.89 

-0.36 

10.41 

9.41 

1.00 

7.16 

6.40 

0.76 

Currently employed and hourly wage 
above $12 (%) 

51.9 

35.2 

16.6 *** 

37.1 

39.3 

- 2.2 

46.7 

35.6 

1 1 1 *** 

22.7 

17.7 

5.0 

Currently employed and hourly wage 
above $15 (%) 

38.9 

24.8 

14 1 *** 

21.0 

21.9 

-0.9 

21.8 

14.5 

7 2 ** 

5.4 

7.9 

-2.5 

Hours 

Average hours worked per week 

27.7 

23.8 

3.8 ** 

27.0 

24.9 

2.1 

34.9 

30.7 

4.2 ** 

24.3 

21.7 

2.5 

Working full-time 3 (%) 

56.7 

46.1 

10.5 ** 

53.0 

49.3 

3.7 

69.6 

60.5 

g i ** 

45.4 

38.3 

7.1 * 

Work schedule and benefits 

Average number of employer-provided 
benefits offered 

2.3 

1.9 

0.4 * 

2.0 

2.1 

- 0.1 

3.1 

2.6 

0.4 * 

2.0 

1.8 

0.2 

Offered health plan or medical insurance 
through employer (%) 

39.3 

33.6 

5.6 

34.3 

35.6 

-1.3 

58.3 

48.9 

g 3 ** 

35.6 

33.0 

2.6 

Obtained a unionized job (%) 

10.5 

9.9 

0.6 

17.0 

16.3 

0.7 

2.6 

2.8 

- 0.1 

6.9 

8.3 

-1.4 

Working regular shift b (%) 

59.9 

46.7 

13 2 *** 

51.7 

47.5 

4.2 

62.7 

60.6 

2.2 

52.9 

41.4 

11.6 *** 


(continued) 
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Appendix Table F.l (continued) 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Very or somewhat satisfied with current 

job 

61.6 

48.3 

13.3 *** 

53.8 

51.8 

2.1 

66.5 

54.7 

1 1 g *** 

46.6 

46.0 

0.6 

Advancement 

Had a job before RA and employed at time 
of survey interview and hourly wage 
increased by $ 8.00 or more 16.1 

7.8 

g 3 *** 

7.8 

7.9 

- 0.1 

9.8 

6.1 

3.7 

1.5 

2.5 

- 1.0 

Job skills at current job 

Scope of work increased 

42.2 

29.0 

13 2 *** 

27.3 

29.7 

-2.5 

41.4 

39.0 

2.5 

33.6 

23.3 

10.3 *** 

Job title changed since job started 

16.0 

11.7 

4.2 

12.0 

13.7 

- 1.8 

18.4 

22.3 

-3.9 

12.4 

12.7 

-0.3 

Offered many opportunities for career 
advancement 

56.4 

41.8 

14 5 *** 

49.3 

47.8 

1.5 

59.1 

48.6 

10.5 ** 

43.9 

34.6 

g 2 ** 

Obtained new skills while working job 

59.9 

46.7 

13 2 *** 

49.6 

38.3 

11.3 ** 

56.2 

46.9 

g 3 ** 

43.4 

37.1 

6.3 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall- Young adjusted p-values were used for 
categorical adjustment. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. No statistically significant differences were found. 

Sample sizes may vary because of missing values. 

a Full-time employment is defined as working 35 hours per week or more. 
b A regular shift includes those worked in the daytime, evening, or nighttime. 



The WorkAdvance Study 
Appendix Table F.2 

Additional Year 2 Impacts on Employment and Material Hardship, by Site 

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment 

Difference Difference Difference Difference 

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) 


Average number of months employed 


or in a skills training program 

13.7 

12.3 

j 4 *** 

13.3 

12.1 

1.2 ** 

14.0 

13.5 

0.5 

13.2 

12.9 

0.3 

All jobs 

Average number of jobs obtained, among 
those employed since RA 

2.0 

1.9 

0.1 

2.2 

1.9 

0.2 

2.0 

2.2 

-0.2 

2.0 

2.0 

0.0 

One job (%>) 

42.9 

38.5 

4.4 

35.1 

38.3 

-3.2 

38.1 

33.5 

4.6 

36.8 

38.0 

-1.1 

Two jobs 

27.1 

34.5 

-7.4 

25.5 

32.5 

-7.0 

32.2 

34.4 

-2.2 

31.1 

28.6 

2.4 

Three jobs 

16.0 

13.4 

2.7 

20.2 

14.9 

5.4 

18.7 

14.5 

4.2 

14.2 

11.6 

2.6 

Four or more jobs 

8.5 

4.0 

4.5 

9.3 

5.6 

3.7 

7.8 

12.6 

-4.7 

7.4 

9.8 

-2.4 

Reasons for not working currently or 
not having worked since RA (%) 

Injury, illness, or disability 

7.6 

8.1 

-0.6 

6.5 

10.5 

-4.0 

13.1 

12.8 

0.4 

17.7 

16.9 

0.8 

Pregnancy, childbirth, or family 
responsibilities 

12.2 

1.1 

11.1 

14.1 

6.4 

7.7 

16.8 

6.6 

10.2 

9.4 

8.4 

1.0 

Going to school/attending training 

4.9 

24.6 

-19.7 

7.5 

4.3 

3.2 

6.2 

9.6 

-3.4 

8.2 

8.6 

-0.4 

Can 't find a job, or looking for a 
job/better job 

34.3 

33.7 

0.6 

28.8 

40.5 

-11.7 

21.6 

35.4 

-13.8 

36.3 

30.4 

5.9 

Transportation problems 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.7 

1.2 

3.6 

2.1 

0.8 

1.3 

Laid off 

8.7 

9.4 

-0.7 

9.8 

14.9 

- 5.0 

14.8 

11.2 

3.5 

10.8 

16.0 

-5.3 

Incarcerated 

0.0 

0.0 

0.0 

2.1 

0.9 

1.2 

2.8 

1.6 

1.3 

0.9 

0.1 

0.9 

Temporary, seasonal, or intermittent 
job completed 

9.4 

11.0 

-1.6 

12.4 

3.5 

8.9 

2.4 

6.1 

-3.6 

3.3 

4.1 

-0.8 

Never had a job 

-I 0.1 

1.2 

-1.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Other 

23.1 

10.9 

12.1 

18.9 

17.3 

1.6 

17.5 

15.6 

1.9 

11.3 

14.7 

-3.4 

Current or most recent job 

Average number of months at job 

6.8 

6.8 

0.0 

6.6 

6.5 

0.1 

7.2 

7.2 

0.0 

5.9 

7.0 

-1.0 * 

Work out-of-state (%) 

8.9 

7.7 

1.2 

7.1 

7.5 

-0.4 

8.9 

5.5 

3.4 

2.0 

2.6 

-0.6 
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Appendix Table F.2 (continued) 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Employer size (%) 

1-9 employees 

14.4 

18.6 

-4.2 

16.3 

16.4 

0.0 

14.4 

17.9 

-3.5 

17.2 

19.4 

-2.2 

10-49 employees 

24.0 

37.8 

-13.8 

23.2 

31.4 

-8.2 

31.6 

27.7 

3.9 

28.8 

32.8 

-4.0 

50-99 employees 

10.5 

10.3 

0.1 

25.0 

11.9 

13.0 

16.0 

9.7 

6.3 

9.9 

9.4 

0.5 

100-499 employees 

25.0 

18.2 

6.8 

23.9 

23.2 

0.7 

20.4 

25.2 

-4.8 

25.8 

21.7 

4.2 

500 or more employees 

26.2 

15.1 

11.1 

11.6 

17.1 

-5.5 

17.6 

19.5 

-1.9 

18.4 

16.8 

1.6 

Change in hours worked per week since 
Did not have a job before RA and/or 
not employed at survey interview" 1 

RA (%) 

37.4 

42.4 

-5.1 

45.6 

40.6 

tt 

5.0 

27.8 

31.0 

-3.2 

45.2 

48.3 

-3.1 

Had a job before RA and employed 
at survey interview and: 

Hours worked increased by: 

20 hours or more 

11.9 

8.8 

3.0 

13.8 

5.6 

8.3 ** 

17.6 

14.6 

3.0 

10.4 

6.9 

3.5 

10 to 19 hours 

10.6 

11.8 

-1.2 

8.2 

10.4 

-2.2 

14.2 

13.7 

0.5 

10.6 

11.0 

-0.4 

5 to 9 hours 

5.4 

5.9 

-0.5 

7.7 

7.1 

0.6 

9.7 

6.2 

3.5 

4.2 

4.4 

-0.2 

Hours worked were within 5 hours 
of hours worked at baseline 

20.7 

18.7 

2.0 

16.8 

19.3 

-2.6 

18.0 

16.4 

1.6 

17.4 

14.6 

2.8 

Hours worked decreased 

14.2 

12.4 

1.7 

7.9 

17.0 

.9 i ** 

12.7 

18.1 

-5.4 

12.2 

14.8 

-2.6 

Benefits at current or most recent job 

Employed at a unionized job (%) 

10.6 

15.1 

-4.5 

20.5 

19.6 

0.9 

2.6 

3.2 

-0.5 

7.9 

9.5 

-1.6 

Average number of employer-provided 
benefits offered 

2.5 

2.2 

0.3 

2.2 

2.3 

-0.1 

3.4 

3.0 

0.4 

2.4 

2.2 

0.2 

Main reason for not enrolling in the 
employer-provided health plan (%>) 
Covered by Medicaid 

4.7 

16.4 

-11.7 

31.1 

32.2 

-1.0 

2.2 

1.4 

0.8 

21.4 

31.1 

-9.7 

Covered by another insurance plan 

36.8 

31.6 

5.2 

11.2 

28.3 

-17.1 

27.4 

33.1 

-5.7 

30.1 

18.6 

11.5 

The cost was too high 

34.8 

20.5 

14.3 

31.1 

12.9 

18.2 

35.9 

26.5 

9.5 

17.4 

26.0 

-8.6 

Had not worked long enough at 
employer 

12.6 

18.9 

-6.3 

25.9 

20.9 

5.0 

26.1 

19.9 

6.3 

21.6 

10.9 

10.7 

Some other reason 

11.1 

12.6 

-1.4 

0.7 

5.8 

-5.0 

8.3 

19.2 

-10.9 

9.5 

13.4 

-3.9 
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Appendix Table F.2 (continued) 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Material hardship (%) 

Experienced the following material 
hardships: b 

Did not pay the full amount of rent 
or mortgage 

19.1 

23.2 

-4.1 

30.3 

34.7 

-4.4 

28.0 

29.0 

-1.0 

32.2 

32.8 

-0.5 

Evicted from home or apartment 

0.9 

2.8 

-1.9 

6.9 

7.4 

-0.5 

4.8 

5.3 

-0.5 

6.7 

4.7 

2.1 

Did not pay the full amount for gas, 
oil, or electricity 

9.2 

16.9 

'J *** 

15.1 

18.9 

-3.8 

26.0 

24.9 

1.1 

33.3 

34.0 

-0.7 

Gas, electrical, or oil services turned 
off or not delivered 

1.9 

4.0 

-2.1 

5.0 

3.9 

1.0 

13.8 

11.7 

2.2 

15.4 

9.3 

6.1 ** 

Phone service disconnected because 
payments were not made 

10.9 

18.4 

-7.5 ** 

21.5 

21.3 

0.2 

19.7 

20.8 

-1.1 

23.7 

23.0 

0.7 

Moved in with family or friends 

16.7 

14.2 

2.5 

21.0 

17.0 

4.0 

20.4 

19.6 

0.8 

14.5 

19.8 

-5.3 

Unable to afford visit to a doctor or 
dentist 

24.7 

28.3 

-3.7 

23.4 

27.9 

-4.4 

37.5 

44.6 

-7.1 * 

32.5 

37.0 

-4.5 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 
for categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

Italic type indicates that the metric is not among the full sample shown in the table. Therefore, the measure is nonexperimental and statistical tests were not 
performed. 

a Sample members who did not work within two years before RA or who were not employed at the time of the survey interview are included in this 
category. 

b With the exception of respondents who indicated being unable to afford visit to a doctor or dentist, all material hardships were reported as having taken 
place within 12 months of the survey interview. 
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The WorkAdvance Study 
Appendix Table F.3 

Year 2 Impacts on Household Composition, Housing Arrangement, and Health Care Coverage, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Marital status 

Single, never married 

75.2 

74.4 

0.8 

62.7 

68.2 

-5.5 

47.9 

50.9 

-3.0 

67.6 

70.4 

-2.8 

Married and living with spouse 

14.5 

17.4 

-2.9 

21.4 

15.3 

6.1 

27.9 

25.6 

2.4 

14.6 

13.4 

1.1 

Married but living apart from spouse 

2.8 

2.7 

0.1 

4.2 

5.4 

-1.2 

3.8 

2.7 

1.0 

3.0 

4.2 

-1.2 

Legally separated, divorced, or 

widowed 

7.6 

5.5 

2.1 

11.7 

11.1 

0.7 

20.5 

20.9 

-0.4 

14.9 

12.0 

2.9 

Living with a partner 

14.2 

11.3 

2.9 

13.7 

20.5 

-6.8 * 

17.0 

20.6 

-3.6 

17.7 

11.6 

6.1 ** 

Housing arrangement in prior month 

Public housing 

5.9 

8.0 

-2.1 

7.9 

12.4 

-4.6 

2.8 

2.3 

0.5 

4.2 

7.0 

-2.8 

Section 8 housing vouchers 

3.2 

4.7 

-1.5 

5.1 

3.7 

1.3 

3.2 

4.1 

-0.9 

10.7 

7.4 

3.3 

Reduced rent 

46.4 

42.6 

3.8 

24.9 

33.9 

-9.0 * 

29.0 

25.7 

3.3 

20.1 

29.3 

_g 2 *** 

Group shelter or homeless 

0.3 

1.2 

-0.9 

4.8 

3.1 

1.7 

0.7 

2.3 

-1.6 

2.3 

1.0 

1.3 

Respondent had health care coverage 

in prior month 3 

65.5 

71.1 

-5.6 

61.9 

66.5 

-4.6 

55.8 

50.9 

4.9 

73.5 

71.0 

2.5 

Public 

29.5 

39.5 

-10.1 ** 

34.6 

40.0 

-5.4 

11.8 

11.6 

0.2 

44.4 

45.0 

-0.6 

Employer-based 

21.1 

16.3 

4.7 

14.5 

17.1 

-2.6 

29.3 

26.5 

2.8 

17.4 

14.0 

3.4 

Other 

16.1 

16.3 

-0.2 

13.7 

10.6 

3.1 

15.7 

13.4 

2.3 

13.4 

13.3 

0.1 

Sample size (total = 2,058) 

287 

265 


205 

179 


297 

263 


286 

276 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall- Young adjusted p-values were used 
for categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: fff = 1 percent; f f = 5 percent; f = 10 percent. No statistically significant differences were found. 

Sample sizes may vary because of missing values. 

“Respondents were allowed to select being covered by more than one type of health insurance, so percentages may sum to more than 100 percent. 



The WorkAdvance Study 
Appendix Table F.4 

Impacts on Employment and Earnings, by Random Assignment Sector, 

at Towards Employment 

Full Sample Health Care Manufacturing 

Difference Difference Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 









All jobs (%) 

Ever employed 

89.2 

87.5 

1.6 

89.6 

87.9 

1.7 

88.5 

87.3 

1.3 


Currently employed 

64.4 

58.2 

6.2 

66.0 

59.8 

6.2 

63.0 

56.1 

6.9 


Percentage of months 
employed 11 

52.5 

57.5 

-5.1 * 

50.2 

56.5 

-6.3 

55.3 

58.3 

-3.0 


Employed 6 or more 
consecutive months' 1 

65.2 

68.3 

-3.1 

64.0 

70.0 

-6.0 

66.8 

66.1 

0.7 


Current or most recent job 

Employed in 
targeted sector (%) 

50.1 

33.7 

\SA *** 

49.1 

37.5 

11.6 ** 

50.7 

29.8 

20 9 *** 


Average total earnings 
per week ($) 

379 

360 

20 

357 

315 

42 

402 

412 

-10 


Average hourly wage ($) 

10.11 

9.78 

0.33 

10.39 

9.19 

1.21 

9.81 

10.45 

-0.64 

t 

Average hours worked 
per week 

33.1 

31.6 

1.5 

31.0 

29.7 

1.3 

35.2 

34.0 

1.2 


Sample size (total = 562) 

286 

276 


149 

146 


137 

130 



Among full research sample. Ul-covered jobs 








Year 1 

Ever employed (%) 

83.6 

78.8 

4.8 * 

82.1 

80.4 

1.7 

84.2 

78.2 

6.0 


Average quarterly 
employment (%) 

61.9 

61.4 

0.5 

60.7 

60.27 

0.4 

62.7 

63.1 

-0.3 


Earnings ($) 

9,500 

9,478 

23 

8,228 

8,218 

9 

10,597 

10,948 

-351 


Year 2 

Ever employed (%) 

79.1 

73.7 

5.4 * 

75.8 

74.1 

1.7 

82.1 

73.4 

8.6 ** 


Average quarterly 
employment (%) 

67.6 

62.5 

5.1 * 

64.8 

65.41 

-0.6 

69.9 

60.1 

g g ** 

t 

Earnings ($) 

13,228 

11,596 

1,632 * 

11,472 

10,526 

946 

14,771 

12,900 

1,871 


First quarter of Year 3 

Ever employed (%) 

68.9 

64.1 

4.8 

72.3 

61.8 

10.5 ** 

65.1 

66.9 

-1.8 

t 

Earnings ($) 

3,318 

3,142 

176 

3,180 

2,777 

403 

3,448 

3,528 

-81 


Sample size (total = 698) 

349 

349 


173 

178 


176 

171 




(continued) 
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Appendix Table F.4 (continued) 


SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 
insurance (UI) administrative records from the Ohio Department of Jobs and Family Services. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 
indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

a Measures reported in time intervals cover only the first 18 months following each sample member's month of 
random assignment (the common follow-up period). 
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The WorkAdvance Study 
Appendix Table F.5 

Quarters 2 to 10 Impacts on Employment Stability and Duration, by Site 




Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Number of quarters worked 

5.5 

5.2 

0.4 * 

5.1 

5.0 

0.1 

6.3 

6.1 

0.2 

5.9 

5.6 

0.3 

Employed entire follow-up period (%) 

18.9 

19.9 

-1.0 

15.1 

20.9 

-5.9 * 

36.8 

34.4 

2.4 

31.8 

31.2 

0.6 

Average number of employers 

2.2 

2.0 

0.2 * 

2.6 

2.2 

0.4 ** 

3.1 

3.2 

0.0 

2.9 

2.8 

0.1 

Number of employers (%) 

Never employed 

11.4 

14.2 

-2.8 

12.4 

16.5 

-4.1 

7.2 

7.2 

0.0 

7.5 

13.4 

tt 
-5.9 ** 

1 to 2 

56.2 

56.3 

-0.1 

42.0 

48.3 

-6.4 

38.7 

39.0 

-0.3 

48.1 

41.4 

6.7 

3 to 4 

24.3 

23.2 

1.1 

32.7 

24.4 

8.4 

30.5 

31.2 

-0.7 

23.7 

25.6 

-2.0 

5 or more 

8.1 

6.3 

1.8 

12.9 

10.8 

2.1 

23.6 

22.6 

1.0 

20.8 

19.6 

1.1 

Average number of quarters in first 
employment spell a 

5.5 

5.1 

0.3 

5.0 

5.0 

-0.1 

6.0 

5.6 

0.3 

5.5 

5.7 

-0.2 

Average first quarter of employment a (#) 

2.5 

2.5 

0.0 

2.3 

2.4 

-0.1 

1.5 

1.7 

-0.2 

2.0 

2.0 

0.1 

Sample size (total = 2,564) 

349 

341 


242 

237 


353 

344 


349 

349 



SOURCES: MDRC calculations from unemployment insurance (UI) administrative records provided by New York State Department of Labor for Per Scholas 
and St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma 
Employment Security Commission for Madison Strategies Group sample members. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 
categorical measures. 

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 
follows: f f f = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

a Measures are among participants who worked in a Ul-covered job within 10 quarters of random assignment. These measures are nonexperimental and 
statistical tests were not performed. 
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The WorkAdvance Study 
Appendix Figure F.l 

Year 2 Impacts on Percentage Employed, by Month Relative to Random Assignment and Site 
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Appendix Figure F.l (continued) 
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Month relative to random assignment 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The WorkAdvance Study 
Appendix Figure F.2 

Year 2 Impacts on Average Monthly Earnings, by Month Relative to Random Assignment and Site 
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(continued) 


Appendix Figure F.2 (continued) 



Towards Employment WA group 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 



The WorkAdvance Study 
Appendix Figure F.3 

Year 2 Impacts on Percentage with Hourly Wage of $12 or More, by Month Relative to Random Assignment and Site 
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Appendix Figure F.3 (continued) 


Madison Strategies Group 



Month relative to random assignment 


K> 
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Towards Employment 



Month relative to random assignment 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The WorkAdvance Study 
Appendix Figure F.4 

Impacts on Percentage Employed in an Unemployment Insurance-Covered Job 
in the Targeted Sector, by Relative Quarter, Site, and Random Assignment Sector 

Madison Strategies Group - Transportation and Manufacturing 

80 ■ 

60 ■ 
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Towards Employment - Health Care 
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Quarter relative to random assignment 

Towards Employment - Manufacturing 



Appendix Figure F.4 (continued) 


SOURCES: MDRC calculations from unemployment insurance (Ul) administrative records provided by Ohio 
Department of Job and Family Services for Towards Employment sample members and Oklahoma Employment 
Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

Sectors are defined by the North American Industry Classification System (NAICS) and are linked to 
employers. NAICS codes are not available in the UI records provided for sample members at Per Scholas and St. 
Nicks Alliance. 

Transportation includes NAICS codes starting with 48-49, manufacturing includes NAICS codes starting with 
31-33, and health care includes NAICS codes starting with 62. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The WorkAdvance Study 
Appendix Figure F.5 

Quarters 1 to 10 Impacts on Percentage Receiving Unemployment Insurance Benefits, 

by Relative Quarter and Site 


Per Scholas 



St. Nicks Alliance 



Quarter relative to random assignment 



(continued) 
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Appendix Figure F.5 (continued) 


SOURCES: MDRC calculations from unemployment insurance (UI) benefits records provided by New York State 
Department of Labor for Per Scholas and St. Nicks Alliance sample members and Oklahoma Employment 
Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group. 

UI benefits data are not available for Towards Employment. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Appendix G 


Further Analysis of Sources of Variation in the 
Economic Impacts 




In seeking reasons for the variation of WorkAdvance effects by site, Chapter 6 explores 
employment in the targeted sector and characteristics of those jobs, differences in impacts 
between the early and late cohorts of enrollees, and differences among the participants across 
the sites. Appendix Figure G.l and Appendix Tables G.l through G. 4 support the discussion of 
differences in participants across the sites. 

In terms of sample composition, the analysis in Chapter 6 found that the impacts of 
WorkAdvance are weaker for those who entered the program with more recent employment. 
Based on this result, if one provider served more participants who were recently employed, it 
might be expected that the impacts at that site would be weaker. This appendix explains two 
analyses that were conducted to understand the extent to which the variation in impacts across 
the sites is due to the characteristics of the individuals who were targeted. 


Conditional Subgroup Analysis 

First, a regression model was estimated, which tried to assess whether the variation in site 
impacts is explained by variation in other characteristics. 1 This analysis focused on earnings in 
Year 2 as a key outcome; the results are shown in Appendix Table G.5. The site impacts were 
conditioned on a series of control variables, which assessed whether the impacts varied by labor 
market attachment status, age, gender, previous incarceration or conviction status, and educa- 
tion. The first set of terms in Appendix Table G.5 show relationships between various baseline 
measures and Year 2 earnings. These are not the main estimates of interest in this scenario. The 
key terms of interest in Appendix Table G.5 are the interaction terms in the second part of the 
table, which show how various baseline characteristics moderate the effects of WorkAdvance. 
The first several interaction terms show the site and cohort impacts. These are followed by 
interactions that control for how the impacts of WorkAdvance varied by the characteristics 
mentioned above. The key issue in this analysis is whether site or background characteristics are 
significant after controlling for both factors simultaneously. If the site impacts are significant, it 
suggests that program factors (that is, implementation factors or other aspects of the site that are 
not controlled for in the analysis) are the dominant source of variation. If the background 
characteristics are significant, but the site interactions are insignificant, this implies that the 
composition of the sample, rather than implementation factors, is the key driver of the impacts. 

The results of Appendix Table G.5 indicate clearly that it is the site factors (that is, im- 
plementation factors) that are dominant. After controlling for multiple interactions between 
baseline characteristics and being a WorkAdvance group member, the impacts at Per Scholas 

'This type of analysis is known as a conditional subgroup analysis. This analysis involves a series of re- 
gressions that test for “interactions” between whether someone was assigned to the WorkAdvance group and 
other baseline characteristics (such as age). 
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and for the late cohorts at Madison Strategies Group and Towards Employment are statistically 
significant. The stronger impacts for the semiattached and long-term unemployed seen in 
Chapter 6 are reduced to statistical insignificance, suggesting that those effects are due to site 
factors, rather than simply long-term unemployment. As discussed in the report, this result does 
not take away from the fact that WorkAdvance was able to increase earnings and employment 
for this group, but it does imply that the stronger effects for this group are related to the fact that 
there were more long-term unemployed at the sites with larger impacts. 

One troubling result in Appendix Table G.5 is that after adjusting for site and other 
baseline characteristics, the impacts of WorkAdvance are significantly weaker among the 
previously convicted or incarcerated. Helping this population into work is a priority for work- 
force development professionals, and it is disappointing that WorkAdvance was, on average, 
unable to do so. 


Compositional Alignment Analysis 

A second analysis tried to align the characteristics of the samples across the sites using weights. 
For this exercise, Per Scholas was chosen as the benchmark site because of the larger impacts 
produced by that program. Sample members at the Per Scholas site were more likely to be male, 
Hispanic, and younger, to have no children, to have more education, and to have higher earn- 
ings in the three years before study entry compared with the other sites. Per Scholas sample 
members were also less likely to have been employed at study entry, to have had a previous 
certification in the targeted sector, and to have worked steadily in the three years before study 
entry. 


This analysis used matching methods in order to identify the individuals at Per Scholas 
who had the most similar characteristics to sample members at the other three sites. 2 Impacts 
were then estimated for this group, suggesting what the impacts of WorkAdvance would have 
been had Per Scholas participants shared baseline characteristics more similar to participants at 
the other sites. The same exercise was then run in reverse: A model was built to estimate 


2 This analysis involved several steps. First, a probability model was developed that used baseline variables 
to predict being a participant at St. Nicks Alliance, Madison Strategies Group, or Towards Employment. This 
model was then applied to the Per Scholas site in order to identify the participants who looked most similar to 
participants from the other three sites. The top third of the sample at Per Scholas (in terms of similarity to the 
sample from the other three sites) was selected. This can also be thought of as the top third most “non-Per 
Scholas-like” participants at Per Scholas. Impacts were then estimated for this subgroup. This analysis is 
experimental because it uses only information from baseline survey and administrative records data, which are 
exogenous to the treatment. Further analysis used the assignment probability as a weight and found similar 
results (not shown). The analysis implicitly assumes, however, that the role of characteristics in moderating the 
impacts is the same across sites, which is not necessarily the case. 
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impacts on participants at the other three sites who were most similar on baseline characteristics 
to Per Scholas sample members. The results of this analysis are shown in Appendix Table G.6. 
Overall, there is no clear pattern to the results. The estimated effects on earnings and employ- 
ment are not statistically significant among Per Scholas sample members who are more similar 
to the samples at the other three sites. There is some limited evidence that impacts are stronger 
(although not always statistically significant) for a few measures among sample members at the 
other three sites who have characteristics similar to those at Per Scholas, but the results are 
inconsistent. 


* * * 

The overall assessment from these two analyses is that the differences in impacts across the sites 
were not due to differences in the composition of the samples, but rather to differences in 
implementation or other factors. 


303 



The WorkAdvance Study 


Appendix Figure G.l 

Activity Flow in Targeted Sector, Among Control Group Respondents, by Site 
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St. Nicks Alliance (N = 179) 
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(continued) 
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Appendix Figure G.l (continued) 


Madison Strategies Group (N = 263) 
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39% 

Obtained job in targeted sector 3 


Towards Employment (N = 276) 



Obtained Obtained Obtained job in targeted 

job in job in sector 3 

targeted targeted 
sector 3 sector 3 

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: Sample sizes vary because of missing values. Measures may not sum to 100 percent because of missing 
values. 

a Jobs in targeted sector are control group members’ current or most recent jobs only. 
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The WorkAdvance Study 
Appendix Table G.l 
Targeted Sector Job Characteristics, 

Among Control Group Survey Respondents Employed Since Random Assignment 



Per Scholas 


St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

Targeted Nonsector 
Sector Job Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Job characteristics 

Hourly wage above $15 

49.8 

29.0 

31.6 

28.5 

22.4 

14.6 

13.8 

11.7 

Working full time b 

73.5 

67.5 

82.3 

70.6 

87.5 

80.7 

76.3 

52.9 

Offered health plan or medical insurance 
through employer 

42.1 

44.3 

44.0 

46.1 

68.7 

50.9 

58.6 

40.6 

Working regular shift c 

74.6 

69.0 

59.4 

72.7 

81.7 

83.7 

81.0 

69.0 

Very or somewhat satisfied with job 

81.1 

66.6 

69.2 

75.7 

74.0 

72.6 

79.6 

72.0 

Advancement 

Had a job before RA and employed at time 
of survey interview and hourly wage 
increased by $8. 00 or more 

27.0 

11.5 

12.6 

15.1 

12.6 

5.4 

5.4 

4.9 

Job skills at current or most recent job 

Scope of work increased 

52.8 

40.5 

42.8 

45.2 

54.5 

49.0 

37.7 

40.7 

Offered many opportunities for career 
advancement 

62.7 

57.0 

55.7 

69.1 

60.1 

58.6 

53.7 

54.1 

Obtained new skills while working job 

69.8 

66.9 

62.2 

61.9 

68.6 

56.9 

66.7 

62.5 

Sample size (total = 892) 

52 

186 

34 

130 

128 

120 

90 

152 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment. 

Statistics are among control group respondents who indicated that they had worked for pay since random assignment. Job characteristics refer to the current 
or most recent job. 

Sample sizes may vary because of missing values. 

a Nonsector job is any job that is not in the sector(s) targeted by the WorkAdvance provider. 
b Full time is considered working 35 hours or more per week. 

C A regular shift includes those worked in the daytime, evening, or nighttime. 
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The WorkAdvance Study 
Appendix Table G.2 


Adjusted Versus Unadjusted Differences in Targeted Sector Versus Non-Targeted Sector Employment 
and Advancement, Among WorkAdvance Group Survey Respondents Employed Since Random Assignment 



Per Scholas 

St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Adjusted 

Unadjusted 

Job characteristics 

Hourly wage above $15 

22.1 

25.0 

21.2 

19.2 

10.3 

15.2 

2.7 

3.0 

Working full time a 

15.9 

16.3 

14.3 

13.1 

28.5 

30.5 

27.0 

27.7 

Offered health plan or medical insurance 
through employer 

15.6 

15.7 

5.4 

-0.2 

36.7 

36.6 

25.1 

26.7 

Working regular shift b 

6.8 

9.0 

3.6 

1.9 

11.6 

9.6 

18.1 

18.3 

Very or somewhat satisfied with job 

13.6 

14.6 

26.6 

22.5 

10.3 

12.0 

2.7 

1.9 

Advancement 

Had a job before RA and employed at time 
of survey interview and hourly wage 
increased by $8.00 or more 

12.2 

12.1 

24.7 

20.5 

0.6 

-1.4 

2.1 

1.9 

Job skills at current or most recent job 

Scope of work increased 

17.2 

19.4 

19.4 

16.7 

20.6 

19.2 

27.2 

25.9 

Offered many opportunities for career 
advancement 

3.9 

6.4 

8.1 

2.4 

29.0 

28.8 

13.0 

12.6 

Obtained new skills while working job 

17.5 

22.6 

19.1 

19.9 

21.2 

19.9 

14.7 

15.6 

Sample size (total = 992) 

272 


183 


285 


252 



SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: RA = random assignment. 

Statistics are among WorkAdvance (program) group survey respondents who indicated that they had worked for pay since random assignment. Job 
characteristics refer to the current or most recent job. 

Sample sizes may vary because of missing values. 

a F li 1 1 time is considered working 35 hours or more per week. 

b A regular shift includes those worked in the daytime, evening, or nighttime. 
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The WorkAdvance Study 
Appendix Table G.3 


Targeted Sector Job Characteristics, 

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment with High Predicted Earnings 



Per Scholas 


St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

Targeted Nonsector 
Sector Job Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job a 

Job characteristics 

Hourly wage above $15 

62.7 

40.3 

48.2 

19.7 

32.5 

29.5 

16.0 

14.4 

Working full time b 

79.2 

61.4 

88.1 

80.0 

95.3 

70.2 

82.0 

61.8 

Offered health plan or medical insurance 
through employer 

54.4 

34.8 

64.4 

46.1 

87.2 

53.4 

69.4 

47.7 

Working regular shift c 

84.0 

76.8 

71.6 

81.5 

85.1 

64.3 

88.2 

72.9 

Very or somewhat satisfied with job 

88.4 

72.7 

96.6 

68.7 

85.9 

87.1 

75.7 

68.1 

Advancement 

Had a job before RA and employed at time 
of survey interview and hourly wage 
increased by $8.00 or more 

32.1 

21.2 

59.7 

-0.5 

14.2 

17.8 

2.0 

-0.3 

Job skills at current or most recent job 

Scope of work increased 

63.7 

52.3 

61.4 

38.8 

53.8 

43.0 

64.2 

17.2 

Offered many opportunities for career 
advancement 

75.9 

69.7 

73.4 

60.4 

79.5 

55.8 

75.4 

52.6 

Obtained new skills while working job 

85.1 

70.0 

84.5 

71.6 

76.7 

65.4 

66.4 

60.7 

Sample size (total = 522) 

132 

61 

27 

45 

101 

39 

70 

47 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance; RA = random assignment. 

Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment with predicted earnings in the 
first quarter of Year 3 above the 50th percentile. Job characteristics refer to the current or most recent job. 

Sample sizes may vary because of missing values. 

“Nonsector job is any job that is not in the sector(s) targeted by the WorkAdvance service provider. 
b Full time is considered working 35 hours or more per week. 

C A regular shift includes those worked in the daytime, evening, or nighttime. 
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Appendix Table G.4 


Targeted Sector Job Characteristics, 

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment with Low Predicted Earnings 



Per Scholas 


St. Nicks Alliance 

Madison Strategies Group 

Towards Employment 

Outcome (%) 

Targeted Nonsector 
Sector Job Job 3 

Targeted 

Sector Job 

Nonsector 

Job a 

Targeted 

Sector Job 

Nonsector 

Job 3 

Targeted 

Sector Job 

Nonsector 

Job 3 

Job characteristics 

Hourly wage above $15 

42.5 

16.0 

48.9 

23.0 

20.7 

11.0 

9.4 

2.5 

Working full time b 

78.9 

59.7 

88.0 

67.6 

96.7 

61.3 

79.3 

42.2 

Offered health plan or medical insurance 
through employer 

48.4 

25.0 

33.4 

35.2 

72.7 

33.9 

51.0 

19.7 

Working regular shift c 

78.3 

58.1 

90.6 

74.2 

84.4 

76.8 

86.6 

68.8 

Very or somewhat satisfied with job 

76.2 

71.8 

90.9 

66.0 

83.9 

63.2 

63.5 

66.1 

Advancement 

Had a job before RA and employed at time 
of survey interview and hourly wage 
increased by $8.00 or more 

25.2 

10.1 

15.6 

9.6 

11.6 

14.3 

11.5 

-0.9 

Job skills at current or most recent job 

Scope o f work increased 

50.8 

21.0 

49.9 

31.5 

61.1 

26.7 

55.1 

40.0 

Offered many opportunities for career 
advancement 

63.6 

59.9 

76.0 

64.6 

77.5 

41.5 

62.1 

55.4 

Obtained new skills while working job 

80.5 

52.3 

84.2 

61.3 

78.3 

48.4 

71.9 

50.4 

Sample size (total = 470) 

45 

34 

39 

72 

88 

57 

73 

62 


SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. 

NOTES: WA = WorkAdvance; RA = random assignment. 

Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment with predicted earnings in the 
first quarter of Year 3 below or equal to the 50th percentile. Job characteristics refer to the current or most recent job. 

Sample sizes may vary because of missing values. 

“Nonsector job is any job that is not in the sector(s) targeted by the WorkAdvance service provider. 
b Full time is considered working 35 hours or more per week. 

C A regular shift includes those worked in the daytime, evening, or nighttime. 



The WorkAdvance Study 
Appendix Table G.5 


Conditional Subgroup Impacts on Year 2 Earnings 


Characteristic 

Parameter Estimate ($) 

Baseline covariates 

Female 

-2,714.10 

*** 

Black 

-1,676.82 

** 

Hispanic 

-16.07 


Age 

-122.51 

*** 

Under age 24 

-1,296.70 


Number of children living with respondent 

-164.60 


Born in state 

606.13 


Highest level of education 

852.25 

*** 

Already has a license/certificate in targeted sector 

877.14 


Currently employed at baseline 

2,786.69 

*** 

Monthly family income 

1.54 


Employed in at least 1 quarter in year prior to RA 

1,096.10 


Employed in all 4 quarters in year prior to RA 

101.45 


Number of quarters employed in 3 years prior to RA 

-213.53 


Total earnings in quarter prior to RA 

0.17 


Total earnings in 3 years prior to RA 

0.15 


Randomly assigned in 201 1 

-3,317.77 

*** 

Randomly assigned in 2012 

-1,793.81 

*** 

Sample member at Per Scholas 

-1,071.84 


Sample member at Towards Employment 

-1,177.76 


Sample member at Madison Strategies Group 

-64.10 


Interactions with research group status 

WorkAdvance group member at Per Scholas 

3,260.68 

** 

WorkAdvance group member at Towards Employment 

-309.50 


WorkAdvance group member at Madison Strategies Group 

451.61 


WorkAdvance group member in late cohort at Per Scholas 

924.56 


WorkAdvance group member in late cohort at Towards Employment 

3,081.64 

* 

WorkAdvance group member in late cohort at Madison Strategies Group 

3,141.66 

** 

WorkAdvance group member and long-term unemployed 

810.47 


WorkAdvance group member and semiattached to labor market 

581.73 


WorkAdvance group member and female 

240.80 


WorkAdvance group member and under age 24 

-887.16 


WorkAdvance group member and previous convicted or incarcerated 

-2,204.29 

** 

WorkAdvance group member and at least some college 

577.82 


R-squared 

0.1705 


F statistic 

14.81 


P-value of F statistic 

<0.0001 



(continued) 
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Appendix Table G.5 (continued) 


SOURCE: MDRC calculations from the WorkAdvance baseline information form and unemployment 
insurance administrative records from New York State Department of Labor, Ohio Department of Jobs 
and Family Services, and Oklahoma Employment Security Commission. 

NOTES: RA = random assignment. 

The late cohort includes sample members randomly assigned in or after Quarter 4, 2012. 

The semiattached subgroup consists of sample members who were unemployed for one to six months 
before random assignment. The long-term unemployed subgroup consists of sample members who have 
never been employed or who were unemployed for seven or more months before random assignment. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The WorkAdvance Study 
Appendix Table G.6 


Quarters 2 to 10 Impacts on Unemployment Insurance-Covered Employment and Earnings 
for Sample Members Demographically Similar to Per Scholas Sample Members 



All Per Scholas 

Sample Members 

Per Scholas Sample Members 
Most Similar to Sample 

Members at Other Sites a 

All Sample Members 
Except Those 

at Per Scholas 

Non-Per Scholas Sample 
Members Most Similar to 

Per Scholas Sample Members* 3 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Year 1 (Ouarters 2 to 5) 

Ever employed (%) 

77.1 

71.0 

6.1 * 

79.0 

69.6 

9.4 

84.4 79.5 

^ g *** 

85.7 

76.3 

Q ^ *** 

UI earnings ($) 

8,868 

8,718 

150 

7,577 

9,095 

- 1,518 

10,813 10,867 

-55 

11,464 

10,429 

1,035 

Year 2 (Ouarters 6 to 9) 

Ever employed (%) 

82.4 

76.4 

6.0 ** 

79.7 

76.9 

2.8 

78.3 76.7 

1.7 

77.8 

76.1 

1.7 

UI earnings ($) 

18,217 

14,471 

3,747 *** 

16,288 

14,093 

2,194 

14,731 13,534 

1,196 * 

15,888 

13,987 

1,901 

First auarter of Year 3 

Ever employed (%) 

70.4 

64.3 

6.0 * 

64.0 

57.8 

6.2 

65.1 63.1 

2.1 

61.9 

62.4 

- 0.4 

UI earnings ($) 

5,385 

4,096 

1,289 *** 

4,516 

3,673 

844 

3,661 3,534 

128 

3,864 

3,842 

22 

Sample size 

349 

341 


108 

126 


944 930 


309 

328 



SOURCES: MDRC calculations from unemployment insurance (UI) administrative records from New York State Department of Labor, Ohio Department 
of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

“This panel shows impacts among the top third of the sample in terms of similarity with sample members at Madison Strategies Group, Towards 
Employment, and St. Nicks Alliance. 

b This panel shows impacts among the top third of the sample in terms of similarity with Per Scholas sample members. 



The WorkAdvance Study 
Appendix Table G.7 


Impacts on Subgroups Defined by Age, All Sites Combined 




Full sample 


18 to 24 


25 to 34 


35 or older 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 
(Impact) Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 70.6 

65.2 

5 3 *** 

72.2 

67.8 

4.5 

75.9 

66.3 

g fi *** 

65.2 

62.9 

2.3 

Weekly earnings ($) 

502 

459 

43 *** 

464 

436 

28 

547 

477 

fig *** 

482 

464 

18 

Sample size (total = 2,058) 

1,075 

983 


265 

228 


361 

366 


449 

389 


Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

82.4 

77.2 

5 2 *** 

88.3 

82.6 

5.7 ** 

82.7 

78.1 

4.6 * 

78.7 

73.4 

5.3 ** 

UI earnings ($) 

10,295 

10,284 

11 

9,535 

9,144 

392 

10,931 

10,932 

-2 

10,180 

10,406 

-226 

Year 2 

Ever employed in a Ul-covered job (%) 

79.5 

76.5 

2.9 * 

85.8 

81.6 

4.2 

79.7 

80.5 

- 0.8 

75.6 

70.1 

5.4 ** 

UI earnings ($) 

15,713 

13,744 

1,969 *** 

13,940 

12,296 

1,644 * 

17,792 

14,750 

3,042 *** 

14,931 

13,838 

1,093 

First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

66.7 

63.3 

3.4 * 

72.8 

63.2 

9.6 ** 

66.4 

67.6 

- 1.2 

63.5 

59.2 

4.3 f 

UI earnings ($) 

4,143 

3,667 

47g H=H=H= 

3,933 

3,098 

835 *** 

4,645 

4,017 

628 ** 

3,848 

3,696 

152 

Sample size (total = 2,564) 

1,293 

1,271 


320 

291 


433 

458 


540 

522 



SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: f ff = 1 percent; f f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 
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The WorkAdvance Study 
Appendix Table G.8 


Impacts on Subgroups Defined by Level of Education, All Sites Combined 




Full sample 

High school, GED, or less 

Some college 

or more 


Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 

70.6 

65.2 

5 ^ *** 

66.9 

61.6 

5.3 * 

73.3 

68.1 

5.2 ** 


Weekly earnings ($) 

502 

459 

43 *** 

467 

435 

32 

528 

478 

50 *** 


Sample size (total = 2,058) 

1,075 

983 


462 

429 


610 

553 



Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

82.4 

77.2 

5 2 *** 

81.7 

74.4 

7 4 *** 

83.0 

79.4 

3.6 * 


UI earnings ($) 

10,295 

10,284 

11 

9,414 

9,102 

312 

10,948 

11,225 

-277 


Year 2 

Ever employed in a Ul-covered job (%) 

79.5 

76.5 

2.9 * 

77.7 

75.6 

2.0 

81.0 

77.0 

4.0 * 


UI earnings ($) 

15,713 

13,744 

1,969 *** 

13,694 

12,426 

1,268 

17,221 

14,816 

2,405 *** 


First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

66.7 

63.3 

3.4 * 

63.5 

61.4 

2.1 

69.1 

64.8 

4.3 * 


UI earnings ($) 

4,143 

3,667 

47g *** 

3,594 

3,392 

202 

4,570 

3,879 

69i *** 


Sample size (total = 2,564) 

1,293 

1,271 


564 

554 


729 

717 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group; GED = General Educational Development certificate. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 
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The WorkAdvance Study 
Appendix Table G.9 


Impacts on Subgroups Defined by Predicted Earnings in First Quarter of Year 3, All Sites Combined 




Full sample 

Less than 25th percentile 

25th to 75th percentile 

Greater than 75th percentile 

Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 
(Impact) Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 70.6 

65.2 

5 3 *** 

54.9 

48.1 

6.9 

71.6 

65.2 

6.3 ** 

83.9 

81.1 

2.7 

Weekly earnings ($) 

502 

459 

43 *** 

358 

367 

-9 

501 

446 

55 *** 

641 

574 

66 ** 

Sample size (total = 2,058) 

1,075 

983 


274 

228 


526 

502 


275 

253 


Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

82.4 

77.2 

5 2 *** 

75.2 

68.1 

7 ^ ** 

81.2 

77.6 

3.7 * 

92.2 

85.4 

6.8 *** 

UI earnings ($) 

10,295 

10,284 

11 

5,997 

5,885 

113 

9,616 

9,572 

43 

16,147 

15,905 

242 

Year 2 

Ever employed in a Ul-covered job (%) 

79.5 

76.5 

2.9 * 

70.7 

60.9 

g g *** 

78.8 

79.4 

-0.7 

90.2 

85.4 

4.8 * ff 

UI earnings ($) 

15,713 

13,744 

1,969 *** 

8,256 

7,833 

423 

14,527 

12,784 

1,743 ** 

25,733 

21,449 

4,284 *** f 

First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

66.7 

63.3 

3.4 * 

57.2 

48.5 

g 7 ** 

64.5 

64.4 

0.1 

81.0 

75.0 

6.0 * 

UI earnings ($) 

4,143 

3,667 

47g H=H=* 

2,157 

1,966 

191 

3,832 

3,428 

403 * 

6,729 

5,881 

848 ** 

Sample size (total = 2,564) 

1,293 

1,271 


334 

307 


639 

643 


320 

321 



SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (Ul) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 
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The WorkAdvance Study 
Appendix Table G.10 


Impacts on Subgroups Defined by Level of Earnings in Three Years Before Study, All Sites Combined 




Full sample 

Less than $10,000 

$10,000-30,000 

$30,000 or more 


Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 70.6 

65.2 

5 3 *** 

63.1 

53.5 

g *** 

71.9 

63.9 

8.0 * 

79.6 

82.4 

- 2.8 

ft 

Weekly earnings ($) 

502 

459 

43 *** 

434 

420 

14 

501 

429 

72 *** 

595 

535 

60 ** 


Sample size (total = 2,058) 

1,075 

983 


463 

422 


262 

253 


347 

307 



Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

82.4 

77.2 

5 2 *** 

75.1 

65.7 

g 4 *** 

84.5 

85.2 

-0.7 

91.4 

87.1 

4.3 ** 

ft 

UI earnings ($) 

10,295 

10,284 

11 

6,649 

6,524 

125 

10,589 

10,116 

473 

15,235 

15,835 

-600 


Year 2 

Ever employed in a Ul-covered job (%) 

79.5 

76.5 

2.9 * 

69.5 

66.4 

3.1 

84.3 

82.6 

1.8 

89.9 

86.1 

3.8 


UI earnings ($) 

15,713 

13,744 

1,969 *** 

10,795 

9,448 

1,347 * 

16,011 

12,902 

3,109 *** 

22,588 

20,479 

2,109 * 


First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

66.7 

63.3 

3.4 * 

56.8 

50.3 

6.5 ** 

70.1 

69.6 

0.5 

78.4 

76.3 

2.1 


UI earnings ($) 

4,143 

3,667 

47g H=H=H= 

2,909 

2,428 

480 ** 

4,034 

3,569 

465 

6,031 

5,481 

550 


Sample size (total = 2,564) 

1,293 

1,271 


570 

562 


320 

320 


403 

389 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: fff = 1 percent; f f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 
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The WorkAdvance Study 
Appendix Table G.ll 


Impacts on Subgroups Defined by Previous Conviction or Incarceration Status, All Sites Combined 




Full sample 3 

Previously convicted or incarcerated 

Not previously convicted or incarcerated 





Difference 



Difference 



Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 










Currently employed (%) 

70.5 

65.3 

5 2 *** 

66.4 

62.0 

4.4 

71.8 

66.6 

5.2 ** 


Weekly earnings ($) 

501 

459 

42 *** 

515 

480 

35 

495 

454 

41 ** 


Sample size (total = 2,058) 

1,074 

983 


266 

269 


808 

714 



Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

82.5 

77.3 

5 2 *** 

79.7 

75.2 

4.4 

83.4 

78.2 

5 2 *** 


UI earnings ($) 

10,302 

10,292 

9 

9,264 

8,892 

373 

10,634 

10,877 

-242 


Year 2 

Ever employed in a Ul-covered job (%) 

79.5 

76.6 

2.9 * 

75.1 

73.4 

1.6 

81.1 

77.8 

3.3 * 


UI earnings ($) 

15,713 

13,759 

1,954 *** 

13,140 

12,137 

1,003 

16,566 

14,451 

2,115 *** 


First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

66.7 

63.3 

3.4 * 

56.5 

59.0 

-2.5 

70.3 

65.0 

5.3 ** 

t 

UI earnings ($) 

4,139 

3,672 

457 h=*h= 

3,157 

3,231 

-74 

4,472 

3,857 

615 *** 

t 

Sample size (total = 2,564) 

1,291 

1,270 


335 

359 


956 

911 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 
York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 


NOTES: WA = WorkAdvance (program) group; C = control group. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: f ff = 1 percent; f f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 

a Full sample includes all sample members included in one of the subgroups. Three sample members are missing a previous conviction or incarceration status 
and therefore are not included in the full sample. 
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Appendix Table G.12 


Impacts on Subgroups Defined by Race/Ethnicity, All Sites Combined 



Latino/FIispanic 


White 


Black/African- American 





Difference 



Difference 



Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

WA 

C 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 

77.7 

67.2 

10.5 ** 

75.2 

72.8 

2.3 

67.5 

61.5 

6.0 ** 


Weekly earnings ($) 

526 

463 

63 * 

545 

536 

9 

460 

427 

33 * 


Sample size (total = 1,762) 

168 

172 


192 

176 


557 

497 



Among full research sample 

Year 1 

Ever employed in a Ul-covered job (%) 

81.6 

72.1 

9.5 ** 

83.3 

82.5 

0.9 

80.9 

76.4 

4.5 ** 


UI earnings ($) 

11,309 

9,267 

2,042 * 

12,079 

12,469 

-389 

8,626 

9,156 

-530 

t 

Year 2 

Ever employed in a Ul-covered job (%) 

83.6 

76.7 

7.0 * 

78.2 

81.7 

-3.5 

79.6 

73.6 

6.0 *** 

t 

UI earnings ($) 

18,835 

14,688 

4 147 *** 

17,307 

15,145 

2,162 * 

13,863 

12,185 

1,678 ** 


First quarter of Year 3 

Ever employed in a Ul-covered job (%) 

71.4 

66.4 

5.1 

64.2 

64.8 

- 0.6 

67.4 

61.7 

5.6 ** 


UI earnings ($) 

5,170 

4,135 

1,035 ** 

4,048 

3,858 

190 

3,824 

3,252 

572 *** 


Sample size (total = 2,180) 

211 

216 


235 

233 


655 

630 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from 
New York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

Fifteen percent of sample members (1) identified as non-Flispanic and listed "Asian," "American Indian," or "Other" as their race; (2) answered as 
"multiracial"; or (3) did not provide a race; these sample members are not shown in this table. 

A two-tailed t-test was applied to the differences between outcomes for the WorkAdvance and control groups. Statistical significance levels are indicated 
as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: f f f = 1 percent; j" f = 5 
percent; f = 10 percent. 

Sample sizes may vary because of missing values. 
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Appendix Table G.13 


Impacts for Subgroups Defined by Random Assignment Cohort, 

Per Scholas and St. Nicks Alliance (Training-First Only Sites) Combined 



Full sample 

Early cohort 


Late cohort 


Outcome 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

WA 

C 

Difference 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 

Currently employed (%) 70.2 65.3 4.9 

68.6 

63.2 

5.5 

71.5 

67.8 

3.7 


Weekly earnings ($) 

544 

480 

65 *** 

559 

471 


524 

493 

31 


Sample size (total = 936) 

492 

444 


258 

220 


234 

224 



Among full research sample 

Year 1 

Ever employed in a 

Ul-covered job (%) 77.5 

70.8 

^ 'J sjs** 

74.4 

66.5 

79 ** 

81.1 

76.0 

5.0 


Ul earnings ($) 

9,074 

9,110 

-36 

8,605 

7,744 

861 

9,588 

10,782 

-1,194 

f 

Year 2 

Ever employed in a 
Ul-covered job (%) 

79.2 

74.6 

4.6 * 

78.2 

72.4 

5.9 * 

80.1 

77.4 

2.6 


UI earnings ($) 

16,701 

14,333 

2,368 *** 

15,433 

12,447 

2,986 ** 

17,996 

16,777 

1,220 


First quarter of Year 3 

Ever employed in a 
Ul-covered job (%) 

66.3 

63.8 

2.5 

65.3 

60.0 

5.2 

66.9 

69.1 

- 2.2 


UI earnings ($) 

4,808 

4,095 

713 ** 

4,395 

3,486 

909 ** 

5,214 

4,901 

314 


Sample size (total = 1,169) 

591 

578 


316 

316 


275 

262 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 
insurance (UI) administrative records from the New York State Department of Labor. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 
includes all sample members randomly assigned in or after Quarter 4, 2012. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 
indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 


319 







The WorkAdvance Study 
Appendix Table G.14 


Impacts for Subgroups Defined by Random Assignment Cohort, 

Towards Employment and Madison Strategies Group (Dual-Track Sites) Combined 



Full sample 

Early cohort 

Late cohort 




Difference 


Difference 


Difference 


Outcome 

WA 

C 

(Impact) 

WA 

C (Impact) 

WA 

C 

(Impact) 

Sig. 

Among respondents to the Year 2 Survey 








Currently employed (%) 

70.9 

65.2 

5.6 ** 

64.3 

59.4 4.9 

77.1 

70.5 

6.7 * 


Weekly earnings ($) 

466 

442 

24 

400 

426 -26 

525 

459 

66 *** 

ttf 

Sample size (total = 1,122) 

583 

539 


282 

257 

301 

282 



Among full research sample 









Year 1 

Ever employed in a 










Ul-covered job (%) 

86.5 

82.6 


84.7 

80.9 3.8 

88.3 

84.2 

4.1 


UI earnings ($) 

11,347 

11,238 

109 

9,746 

9,984 -238 

12,936 

12,375 

561 


Year 2 

Ever employed in a 










Ul-covered job (%) 

79.6 

78.2 

1.4 

75.2 

75.2 0.0 

83.9 

81.0 

2.9 


UI earnings ($) 

14,876 

13,258 

1,618 ** 

11,633 

11,803 -170 

18,055 

14,548 

3,507 *** 

tff 

First quarter of Year 3 

Ever employed in a 










Ul-covered job (%) 

66.7 

63.0 

3.7 

60.1 

60.7 -0.6 

73.2 

65.0 

8.2 ** 

f 

UI earnings ($) 

3,577 

3,317 

260 

2,863 

2,950 -87 

4,279 

3,644 

636 ** 

f 

Sample size (total = 1,395) 

702 

693 


341 

341 

361 

352 




SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 
insurance (Ul) administrative records from the Ohio Department of Jobs and Family Services and Oklahoma 
Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group. 

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 
includes all sample members randomly assigned in or after Quarter 4, 2012. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 
indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Sample sizes may vary because of missing values. 


320 







References 


Acs, Gregory, and Pamela Loprest. 2004. “Leaving Welfare: Challenges in a New Economy.” 
Employment Research 11,4: 5-6. 

Bamow, Burt S., and David Greenberg. 2015. “Do Estimated Impacts on Earnings Depend on 
the Source of the Data Used to Measure Them? Evidence from Previous Social 
Experiments.” Evaluation Review 39, 2: 179-228. 

Bloom, Dan, Richard Hendra, and Jocelyn Page. 2006. The Employment Retention and 
Advancement Project: Results from the Chicago ERA Site. New York: MDRC. 

Bloom, Dan, and Charles Michalopoulos. 2001. How Welfare and Work Policies Affect 
Employment and Income: A Synthesis of Research. New York: MDRC. 

Bloom, Howard S., and Charles Michalopoulos. 2010. When Is the Story in the Subgroups? 
Strategies for Interpreting and Reporting Intervention Effects on Subgroups. New York: 
MDRC. 

Bloom, Howard S., Larry L. Orr, Stephen H. Bell, George Cave, Fred Doolittle, Winston Lin, 
and Johannes M. Bos. 1997. “The Benefits and Costs of JTPA Title 11-A Programs: Key 
Findings from the National Job Training Partnership Act Study.” Journal of Human 
Resources 32, 3: 549-576. 

Bureau of Labor Statistics. 2014. Employee Benefits in the United States — March 2014. 
Washington, DC: Bureau of Labor Statistics, U.S. Department of Labor. 

Card, David, Jochen Kluve, and Andrea Weber. 2010. “Active Labour Market Policy 
Evaluations: A Meta- Analysis.” Economic Journal 120, 548: F452-F477. 

Card, David, Jochen Kluve, and Andrea Weber. 2015. “What Works? A Meta Analysis of 
Recent Active Labor Market Program Evaluations.” Institute for the Study of Labor (1ZA) 
Discussion Paper No. 9236. Bonn, Germany: 1ZA. 

Card, David, Charles Michalopoulos, and Philip K. Robins. 2001. “The Limits to Wage 
Growth: Measuring the Growth Rate of Wages for Recent Welfare Leavers.” Working 
Paper 8444, National Bureau of Economic Research. 

Cohen, Jacob. 1992. “A Power Primer .” Psychological Bulletin 112, 1: 155. 

Conway, Maureen, Amy Blair, Steven L. Dawson, and Linda Dworak-Munoz. 2007. Sectoral 
Strategies for Low-Income Workers: Lessons From The Field. Washington, DC: Aspen 
Institute. 

D’Amico, Ronald. 2006. What’s Known About the Effects of Publicly-Funded Employment and 
Training Programs . Washington, DC: Social Policy Research Associates. 

Friedlander, Daniel. 1988. “Subgroup Impacts and Performance Indicators for Selected Welfare 
Employment Programs.” New York: MDRC. 


321 



Ghayad, Rand. 2013. “The Jobless Trap.” Working paper. Boston: Federal Reserve Bank of 
Boston and Northeastern University. 

Gladden, Tricia, and Christopher Taber. 2009. “The Relationship Between Wage Growth and 
Wage Levels.” Journal of Applied Econometrics 24, 6: 914-932. 

Golden, Lonnie. 2015. Irregular Work Scheduling and Its Consequences. Washington, DC: 
Economic Policy Institute. 

Gueron, Judith M., and Gayle Hamilton. 2002. The Role of Education and Training in Welfare 
Reform. New York: MDRC. 

Gueron, Judith M., and Howard Rolston. 2013. Fighting for Reliable Evidence. New York: 
Russell Sage Foundation. 

Hamilton, Gayle, Stephen Freedman, Lisa Gennetian, Charles Michalopoulos, Johanna Walter, 
Diana Adams-Ciardullo, Anna Gassman-Pines, Sharon McGroder, Martha Zaslow, Surjeet 
Ahluwalia, and Jennifer Brooks. 2001. How Effective Are Different Welfare-to-Work 
Approaches? Five-Year Adult and Child Impacts for Eleven Programs. New York: MDRC. 

Hamilton, Gayle, and Susan Scrivener. 2012a. Facilitating Postsecondaiy Education and 
Training for TANF Recipients. Brief No. 07. Washington, DC: Urban Institute. 

Hamilton, Gayle, and Susan Scrivener. 2012b. Increasing Employment Stability and Earnings 
for Low-Wage Workers: Lessons from the Employment Retention and Advancement (ERA) 
Project. OPRE Report 2012-19. Washington, DC: Office of Planning, Research and 
Evaluation, Administration for Children and Families, U.S. Department of Health and 
Human Services. 

Henderson, Kathryn, Crystal MacAllum, and Mustafa Karakus. 2010. Workforce Innovations: 
Outcome Analysis of Outreach, Career Advancement and Sector-Focused Programs. 
Rockville, MD: Westat. 

Hendra, Richard, Keri-Nicole Dillman, Gayle Hamilton, Erika Lundquist, Karin Martinson, and 
Melissa Wavelet. 2010. How Effective Are Different Approaches Aiming to Increase 
Employment Retention and Advancement? Final Impacts for Twelve Models. New York: 
MDRC. 

Hendra, Richard, James A. Riccio, Richard Dorsett, David H. Greenberg, Genevieve Knight, 
Joan Phillips, Philip K. Robins, Sandra Vegeris, and Johanna Walter. 2011. Breaking the 
Low-Pay, No-Pay Cycle: Final Evidence from the UK Employment Retention and 
Advancement (ERA) Demonstration. New York: MDRC. 

Hill, Carolyn J., Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey. 2007. 
Empirical Benchmarks for Interpreting Effect Sizes in Research. New York: MDRC. 

Hilliard, Thomas. 2013. “Building the American Workforce.” Working paper. New York: 
Council on Foreign Relations. 


322 



Hollenbeck, Kevin. 2007. Process and Net Impact Evaluations of the Focus: HOPE Adult 
Training Programs and Student Loan Fund. Upjohn Institute Technical Report No. 07- 
023. Kalamazoo, Ml: W.E. Upjohn Institute for Employment Research. 

Holzer, Harry J. 2010. “Avoiding a Lost Generation: How to Minimize the Impact of the Great 
Recession on Young Workers.” Testimony before the Joint Economic Committee of the 
U.S. Congress. Washington, DC: Urban Institute. 

Holzer, Harry J. 2013. “Skill Mismatches in Contemporary Labor Markets: How Real? And 
What Remedies?” Conference paper. Washington, DC: Atlantic Council and University of 
Maryland School of Public Policy. 

Hotz, V. Joseph, and John Karl Scholz. 2001. “Measuring Employment and Income for Low- 
Income Populations with Administrative and Survey Data.” Discussion Paper 1224-01. 
Madison, Wl: Institute for Research on Poverty. 

Kolesnikova, Natalia, and Yang Liu. 2011. “Jobless Recoveries: Causes and Consequences.” 
Regional Economist, April. Website: www.stlouisfed.org/publications/regional- 
economist. 

Komfeld, Robert, and Howard S. Bloom. 1999. “Measuring Program Impacts on Earnings and 
Employment: Do Unemployment Insurance Wage Reports from Employers Agree with 
Surveys of Individuals?” Journal of Labor Economics 17, 1: 168-197. 

Kroft, Kory, Fabian Lange, and Matthew J. Notowidigdo. 2013. “Duration Dependence and 
Labor Market Conditions: Evidence from a Field Experiment.” Quarterly Journal of 
Economics 128, 1123-1167. 

Lower-Basch, Elizabeth, and Neil Ridley. 2013. Navigating Federal Programs to Build 
Sustainable Career Pathways in the Health Professions: A Guide for HPOG Programs. 
Washington, DC: Center for Law and Social Policy. 

Maguire, Sheila, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz. 2010. 
Tuning In to Local Labor Markets: Findings from the Sectoral Employment Study. 
Philadelphia: Public/Private Ventures. 

Mangat, Ravinder. 2007. Sector Snapshot: A Profile of Sector Initiatives, 2007. Oakland, CA: 
National Network of Sector Partners. 

Mangat, Ravinder. 2010. Sector Snapshot: A Profile of Sector Initiatives, 2010. Oakland, CA: 
National Network of Sector Partners. 

Martinson, Karin, and Karen Gardiner. 2014. Improving the Economic Prospects of Low- 
Income Individuals Through Career Pathways Programs: The Innovative Strategies for 
Increasing Self Sufficiency Evaluation. OPRE Report 2014-17. Washington, DC: Office of 
Planning, Research and Evaluation, Administration for Children and Families, U.S. 
Department of Health and Human Services. 

Mayer, Gerald. 2014. “The Increased Supply of Underutilized Labor from 2006 to 2014.” 
Monthly Labor Review, November. Website: www.bls.gov/opub/mlr. 


323 



Melendez, Edwin. 1996. Working on Jobs: The Center for Employment Training. Boston: 
Mauricio Gaston Institute. 

Miller, Cynthia, Betsy L. Tessler, and Mark van Dok. 2009. Implementation and Early Impacts 
of the Work Advancement and Support Center (WASC) Demonstration. New York: MDRC. 

Miller, Cynthia, Mark van Dok, Betsy L. Tessler, and Alexandra Pennington. 2012. Strategies 
to Help Low- Wage Workers Advance: Implementation and Final Impacts of the Work 
Advancement and Support Center (WASC) Demonstration. New York: MDRC. 

Mishel, Lawrence, Jared Bernstein, and Sylvia Allegretto. 2007. The State of Working America 
2006/2007: An Economic Policy Institute Book. Ithaca, NY : 1LR Press. 

Mishel, Lawrence, Jared Bernstein, and Heidi Shierholz. 2009. The State of Working America: 
2008/2009. Ithaca, NY: 1LR Press. 

Morrison, Tom, Bob Maciejewski, Craig Giffi, Emily Stover DeRocco, Jennifer McNelly, and 
Gardner Carrick. 2011. Boiling Point? The Skills Gap in U.S. Manufacturing. Washington, 
DC: Deloitte and The Manufacturing Institute. 

National Skills Coalition. 2011. Workforce Investment Act, Title I: Workforce Investment 
Systems for Adults, Dislocated Workers, and Youth. Washington, DC: National Skills 
Coalition. Website: www.nationalskillscoalition.org/lnBrief. 

Olken, Benjamin. 2015. “Promises and Perils of Pre -Analysis Plans.” Journal of Economic 
Perspectives 29, 3: 61-80. 

Osterman, Paul, and Brenda Lautsch. 1996. Project QUEST: A Report to the Ford Foundation. 
Cambridge: Massachusetts Institute of Technology, Sloan School of Management. 

Puma, Michael J., Robert B. Olsen, Stephen H. Bell, and Cristofer Price. 2009. What to Do 
When Data Are Missing in Group Randomized Controlled Trials (NCEE 2009-0049). 
Washington, DC: National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, U.S. Department of Education. 

Rangarajan, Anu, and Tim Novak. 1999. The Struggle to Sustain Employment: The 
Effectiveness of the Postemployment Sendees Demonstration. Springfield: Illinois 
Department of Human Services. 

Riccio, James A., Nadine Dechausay, Cynthia Miller, Stephen Nunez, Nandita Verma, and 
Edith Yang. 2013. Conditional Cash Transfers in New York City: The Continuing Story of 
the Opportunity NYC-Family Rewards Demonstration. New York: MDRC. 

Schochet, Peter, and Anu Rangarajan. 2004. Characteristics of Low-Wage Workers and Their 
Labor Market Experiences: Evidence from the Mid- to Late 1990s. Washington, DC: U.S. 
Department of Health and Human Services. 

Tessler, Betsy. 2013. WorkAdvance: Testing a New Approach to Increase Employment 
Advancement for Low-Skilled Adults. New York: MDRC. 


324 



Tessler, Betsy L., Michael Bangser, Alexandra Pennington, Kelsey Schaberg, and Hannah 
Dalporto. 2014. Meeting the Needs of Workers and Employers: Implementation of a 
Sector-Focused Career Advancement Model for Low-Skilled Adults. New York: MDRC. 

Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the 
Royal Statistical Society, Series B (Methodological) 58, 1: 267-288. 

Tompson, Trevor, Jennifer Benz, Jennifer Agiesta, and Dennis Junius. 2013. America ’s Lower- 
Wage Workforce: Employer and Worker Perspectives. Chicago: Associated Press-NORC 
Center for Public Affairs Research. 

U.S. Congress. 2015. “S.1772 - Schedules That Work Act.” Website: 
https://www.congress.gov/bill/114th-congress/senate-bill/1772. 

U.S. Department of Labor. 2015. “What Is the Workforce Innovation Fund?” Website: 
http ://www. doleta. gov/ workforce_inno vation/. 

Van Horn, Carl, Kathy Krepcio, and Maria Heidkamp. 2011. Research Report: Improving 
Education and Training for Older Workers. Washington, DC: AARP Public Policy 
Institute. 

Woolsey, Lindsey, and Garrett Groves. 2013. State Sector Strategies Coming of Age: 
Implications for State Workforce Policymakers . Washington, DC: National Governors 
Association Center for Best Practices. 

Zandniapour, Lily, and Maureen Conway. 2002. Gaining Ground: The Labor Market Progress 
of Participants of Sectoral Employment Development Programs. Sectoral Development 
Learning Project (SEDLP) Research Report No. 3. Washington, DC: Workforce Strategies 
Institute at the Aspen Institute. 


325 



About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



